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STUDY  TITLE: 


Gr.NaHAi*  PUnFCbi  CCkPJT-h  AO^’.JIbiTICN 


STUDY  PROJECT  COALS: 

i Computer  usage  is  pervasive  in  civilian  and  govu roment  sectors.  The  stops  \iu ich 
are  essential  to  t t.o  acquisition  of  a computer  system  must  be  known.  The  real  is 
to  provide  an  overview  of  trio  computer  acquisition  process  with  a discussion  of 
tne  metnods  available  to  a manager  in  the  evaluation  and  selection  of  a computer 
system  or  parts  thereof. 
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STUDY  REPORT  ABSTRACT: 

The  paper  is  divided  into  four  major  chapters,  each  of  which  takes  tne  reader 
through  a eerier  of  steps  which  are  essential  to  the  tack  of  evaluating-  and 
selecting  a computer  system.  Report  concentrates  on  peroral  purpose  commer- 
cially available  (non-embedaed ) computers.  The  inquiry  vac  conducted  by  j:cani 
library  research.  Tne  emphasis  on  topic  eelection  is  that  of  ihe  author  ar.d  \ • 
no  means  either  all-inclusive  or  exhaustive.  The  attempt  was  to  review  current 
li tore  tore  pertaining  to  ADP  acquisition  procosc.  Computers  are  pervasive. 
Project  managers  (PK)  cannot  ignore  tnem.  Typical  mana-ars  prefer  tc  delegate 
tr.a  subject  of  computers  to  specialist  groups  for  detailed  actions.  A mannver 
must  bo  a-nsre  cf  suen  subjects  as  benchma  rkin£,  simulation,  cost-va  )ue/scores~ 
weights  approach,  etc.,  as  they  apply  to  computer  evaluotion/selscticn.  This 
paper  discusses  taese  aspects  und  many  more.  In  essence,  it  serve?  nr.  pr,  over- 
view of  aetnods  in  toe  aoove  subject  area,  whicii  a FM  can  use  to  pick  und  cnco. 
from  a?  nis  particular  rules,  regulations,  and  circumstances  permit. 


ci 

by 


\ 


ADPh 


COKPUTSRS 


C J 


ifCfhSlG*  V 

| HIS 

■ 

11  1 

uw*<*  os’— ra 

C 

□ 

IT 

61$.  VJ*  .i  • 

•-TT  ! JCt 

G- 1 

:u.  ‘l  ; 

NAME . RANK,  SERVICE 

i r« it o • 6 o • fh  i r i jer  | Jw<i 


1 


CLASS 

Tv.C  76-2 


general  purpose  computer  acquisition 


Study  Project  Report 
Individual  Study  Program 


Defense  Systems  Management  College 
Program  Management  Course 
Class  76-2 


by 

Francis  C.  Marr 

Ka  j USA 


November  1976 


Study  Project  Advisor 
LCDH  Susan  Anderson,  USN 


This  study  project  report  represents  tne  views,  conclusions  and  recommen- 
dations of  tne  autnor  and  does  not  necessarily  reflect  the  official  opinion 
of  tne  De:on3e  systems  Management  Collere  or  tne  Department  of  Defense. 


iXiCunvE  sumax 


This  study  provides  an  overview  of  the  Automatic  Data  Processing  (AD?) 
acquisition  process  as  it  relates  to  general-purpose  (non-embedded ) com- 
puter systems.  It  starts  with  a broad  overview  and  proceeds  through  the 
general  selection  process  with  a detailed  discussion  of  techniques  which  are 
available  to  a program  manager  (PM). 

Computers  are  utilized  in  all  walks  of  life  in  both  government  and 
industry.  It  would  be  a rare  system  whicn  did  not  involve  even  a tangen- 
tial relationship  to  a computer  or  its  related  equipment.  A manager  mus-t 
be  aware  of  tne  overall  AD?  selection  and  evaluation  process.  A PM  is 
totally  responsiole  for  tue  management  of  his  project.  While  experts  are 
available  to  assist  him  in  many  areas  he  muBt  not  decide  to  be  unfamiliar 
witn  t;iS  basic  principles  of  any  area. 

This  paper  covers  tne  analysis  and  specification  of  the  user's  system 
and  the  tools  to  aid  in  tns  investigation  such  as  simulation,  modeling, 
monitors,  etc.,  and  some  commercially  available  software  packages.  Valida- 
tion of  tne  proposed  computer  system  is  tne  most  crucial  phase  in  the  entire 
selection  process.  Several  excellent  methods  of  selection  are  provided  in 
the  form  of  the  weighted  scores  approach  and  the  cost-value  technique.  The 
latter  method,  in  particular,  is  very  valuable.  This  procedure  recognizes 
the  necessity  for  evaluating  the  non-mandatory  requirements  of  a system  and 
their  costs.  The  mandatory  requirements  are  validated  instead  of  being 
evaluated • 

It  appears  tnat  tne  major  emphasis  wnich  a manager  must  place  in  tne 
overall  process  is  on  a modification  in  the  initial  analysis  time.  In 


order  to  lessen  tne  subjectivism  whicn  is  already  a part  of  any  evaluation 
and  to  lessen  tne  criticism  of  'representativeness,*  considerably  more  time 
and  effort  must  be  expended  at  tne  'front  end"  of  any  systems  development, 
hany  commercially  available  simulators  must  be  used  in  order  to  evaluate  a 
users  present  or  proposed  system.  Bencnmarks  should  also  be  utilized  even 
though  they  are  not  always  satisfactory. 

In  essence,  the  is  provided  with  an  overview  of  the  evaluation  and 
selection  process,  he  can  be  brought  "up-to-speed*  on  the  subject  area  by 

t 

using  this  paper.  Then,  depending  upon  the  particular  situation,  he  can 
pick  and  choose  the  various  metnods  as  the  circumstances  and  rules  allow. 
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CriAPTiR  I 


INTRODUCTION 

1 .1  General 

This  paper  is  divided  into  four  major  cnapters,  each  of  which  will 
take  the  reader  tnrougn  a series  of  steps  wnich  are  essential  to  the  task 
of  evaluating  and  selecting  a computer  system.  The  emphasis  of  topic 
selection  is  that  of  tne  author  and  is  by  no  means  either  all-inclusive  or 
exhaustive . 

The  researen  question  is  as  follows: 

What  methods  are  available  to  a manager  in  the  evaluation 
and  selection  of  a computer  system  or  parts  thereof? 

1 .2  Purpose  and  Scope  of  Researen 

Tne  purpose  of  this  research  is  twofold.  Tne  first  is  to  satisfy  the 
partial  requirements  for  tne  Program  i-janegers  Course.  The  second  purpose 
is  my  modest  attempt  to  learn  more  about  tne  evaluation  and  selection  of  a 
computer  system.  Tnis  paper  is  directed  towards  tne  general  purpose  com- 
mercially available  (non-embedded ) computer  systems.  If  tnrougn  this 
paper  a project  manager  can  more  accurately  determine  a procedure  which  is 
useful  when  procurement  of  ADP  is  required  for  successful  project  comple- 
tion, tnen  tnis  research  will  serve  seme  useful  purpose. 

The  scope  of  this  research  is  limited  to  a general  description  of 
computer  acquisitions  with  special  emphasis  on  benchmarking,  simulation, 
partial  testing — in  snort,  the  workload  description  and  validation  pro- 
cesses. No  particular  computer  user  group  is  addressed.  If  a vendor  or 
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his  products  are  mentioned,  it  is  done  so  only  in  passing  and  tnon  only  to 
stress  a point  for  discussion. 
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1 Chapter  Summaries 


Chapter  II  provides  an  overview  of  tne  selection  process.  Included 
therein  are  sucn  lectors  as  tne  basic  considerations  in  systems  procurement, 
sola  source  versus  competitive  Did  procedures,  staffing  views,  plus  a few 
otner  comments. 

Chapter  III  deals  with  tne  analysis  and  specification  of  the  user's 
present  system  and  tne  tools  to  aid  in  tnat  investigation  sucn  as  simula- 
tion, modeling,  monitors,  and  some  commercially  available  software  packages. 

( 

It  also  covers  economic  analysis  and  systems  life  evaluation  costing.  The 
section  dealing  witn  tne  validation  of  tne  proposed  system  is  the  most 
crucial  phase  in  tne  entire  selection  process.  Simulation  ia  investigated, 
as  well  as  tne  methods  of  central  processine  unit  (CPU)  timing,  and  soft- 
ware evaluation.  Means  for  benchmarkinr  are  described.  The  last  section 
in  this  chapter  addresses  the  selection  techniques  through  which  the  best 
overall  system  may  be  chosen.  The  weignted  scores  approacn  and  tne  cost- 
value  tecnniques  are  discussed. 

Cnapter  IV  describes  tne  progress  which  nas  bean  made  in  tne  evalua- 
tion end  selection  process. 

Cnapter  V contains  tne  summary.  Tne  tnesis  question  is  addressed  and 
tne  conclusions  of  tnis  study  paper  are  presented. 

Two  clarifying  comments  are  needed.  First,  it  is  hoped  that  tne  re  - 
searcn  will  provide  a broad  introduction  to  the  computer  evaluation  and 
selection  process.  Tne  outline  of  the  cnapters  is  intended  to  portray  the 
fact  tnat  tne  total  scope  of  the  acquisition  process  must  be  known  by  a 
manager.  In  this  paper  tne  words  "organization,*  "agency,"  and 
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corporation"  have  identical  meaning  and  intent.  Partner,  the  term 
system*  has  cany  meanings  and  connotations.  In  this  context  tne  word 
system"  refers  to  tne  data  processing  type  meaning. 


CHAPTER  II 


OVERVIEW 


2.1  Introduction 

Tne  primary  intent  of  section  2.2  is  to  provide  a broad  introduction 
to  AD?  systems  acquisition  with  empnasis  on  the  DCD  policy  pertaining  to 
tnie  subject.  Subsequent  sections  in  this  cnapter  provide  more  overview 
type  details  wnicn  must  oe  considered  by  tne  PM  in  a computer  acquisition. 
Section  2. 3 covers  primarily  tne  past  and  present  analysis  aspects  of  com- 
puter systems  design.  In  addition,  various  past  and  present  methodolgies 
for  testing  tne  newly  designed  system  are  presented — tne  core  current  of 
wnicn  will  be  discussed  in  Chapter  Tne  next  section  reviews  basic  con- 
siderations with  respect  to  sole  source  versus  competitive  acquisition 
processes.  A knowledge  of  tne  basic  difference  between  tnese  two  concepts 
is  a must  for  tne  ?!■'.,  otherwise  he  may  be  tempted  to  jump  at  the  first 
offeror  wno  is  possibly  backed  by  a big  name  company.  As  tne  section 
points  out  competition  is  tne  better  route.  Next  tne  PM  needs  to  consider 
wno  is  going  to  perform  all  of  tne  evaluating  and  selecting  tasks.  Sec- 
tion 2.5  covers  tnese  aspects  in  some  detail  in  order  to  provide  a better 
understanding  tnereof.  Lastly,  Section  2.o  provides  a brief  discussion  of 
probably  tne  most  key  event  in  tne  entire  process.  The  user  must  recognize 
tnat  there  are  both  good  and  bad  suppliers.  Tne  basic  step  to  eliminatinr 
mucn  future  trouole  is  an  adequately  defined  Request  for  Proposal  (RFP). 
while  this  subject  will  be  discussed  later,  it  is  essential  to  bring  out 
certain  salient  overview  facts. 
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2.2  3aalc  Ccnsld era tions  in  Systems  procurement 


A compariaon  ason?  various  computing  systems  available  today  is  be- 
coming more  difficult  since  many  systems  differ  in  size,  configuration,  end 
basic  design.  3efore  the  evaluation  and  selection  process  can  begin,  how- 
ever, some  basic  considerations  must  be  addressed. 

Tnough  rarely  a controlling  factor,  tne  cost  of  the  potential  acquisi- 
tion is  still  a major  factor  and  little  has  to  be  said  to  stress  its  im- 
portance. aitnout  tnis  constraint  little  effort  would  need  to  be  expended 

( 

in  modeling,  simulation,  and  bencnma rking. 

Time  is  an  important  factor  in  most  system  procurements,  besides  the 
fact  tnat  tne  user  would  like  to  nave  nis  new  equipment  as  soon  as  possible, 
it  is  important  tnat  each  phase  of  the  acquisition  be  alloted  enough  time  to 
permit  full  completion  of  its  associated  tasxs.  The  time  allotted  to  each 
pnase  in  an  evaluation  and  selection  process  is  system  dependent.  The  more 
complex  and  costlier  system  must  be  alloted  more  resources,  botn  in  terms 
of  dollars  end  time.  Expenses  for  smaller  procurements  are,  by  necessity, 
curtailed  if  tne  selection  process  is  to  remain  economical. 

Tne  following  table,  taxen  from  the  General  bervices  Administration 
AD?  Procurement  Guidelines,  indicates  tne  various  time  spans  for  the  ele- 
ments of  tne  competitive  procurement  cycle.  Tnis  was  determined  from  a 
recent  interagency  study. 

Heauest  for  Proposal  (nf?)  Development  5 months 

Delegation  of  procurement  authority  from  GbA  2C  days 
dencnmark  time  4c  days 

Tecnnical  evaluation  time  lA  weeks 
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Contract  negotiations 


6 weeks 


Cost  evaluation  time  12  weejcs 

selection  time  6 weeks 

TOTAL  A5  weeks 

The  above  times  are  historically  typical.  Consequently,  tne  total  for 
all  elements  of  the  procurement  cycle  will  not  necessarily  equal  tne  aver- 
age time  for  tnat  category  of  procurement.  It  should  be  noted,  however, 
tnat  a recent  study  by.  .«ir . -.Robert  H.  ?arke  (ri-'.O  Class  76-1)  indicated  tnat 
an  average  processing  cycle  of  6?  weeks  existed  within  the  Department  of 
tne  Army  on  AD?  procurement  actions.  (2C,  19)*  As  a result  one  6hould 
view  tne  procurement  cycle  with  extreme  caution  and  planning  whenever  AD? 
is  involved  in  a program.  In  essence,  one  will  not  just  order  a computer 
today  and  receive  it  tomorrow. 

The  user's  needs  may  change  if  the  acquisition  cycle  is  long  enough. 

In  addition,  staff  salaries,  tr.e  number  cf  system  analysts  and  programmers, 
and  similar  factors  can  profoundly  affect  tne  cost  or  a computer  system 
and  the  productivity  it  provides.  (26,  2C2) 

Alternatives  to  tne  established  plan  should  become  part  of  tne  plan 
and  not  dismissed  out-of-nand.  For  various  reasons  it  may  be  less  costly 
to  lease  a new  system  r«t..er  than  purchase  tne  equipment  outrignt.  Simi- 
larly, tne  entire  exercise  may  prove  tnat  updating  tne  current  system  is 
cheaper.  In  addition,  it  may  prove  tnat  tne  existing  system  will  be 

*Tnis  notation  will  be  used  throughout  tne  report  for  sources  of 
quota ti  ns  and  major  references.  Tne  :irst  number  is  tne  source  listed  in 
tne  bicliograpny . The  second  number  is  tne  page  in  tne  reference. 
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satisfactory  for  X mors  years.  Since  most  system  development  is  approacned 
with  a "let's  dream"  attitude,  alternatives  in  costing  should  be  considered 
such  as  determining  tne  total  cost  of  the  new  system  both  with  and  without 
the  "bells  and  whistles." 

Another  basic  factor  which  oust  be  considered  before  proceeding  with  a 
system  procurement  is  the  impact  of  a new  system  on  the  organization  as  a 
whole.  Literally  hundreds  of  books  and  thousands  of  papers  have  been  writ- 
ten on  the  effects  that  new  influences  have  on  employees  wnen  they  are  taken 
out  of  their  old  environment.  Depending  on  the  complexity  of  the  system  in 
question,  retraining  and  reeducating  must  be  considered.  During  the  acqui- 
sition, seme  departments  may  be  reduced  in  size  wnile  otners  grow.  In 
essence,  tne  total  impact  must  be  known  beforenand. 

Within  tne  Department  of  Defense  (DCD)  policy  and  guidance  for  the 
selection  and  acquisition  of  A D?D  is  contained  in  DCD  Directive  AlCp.55. 

(7,  5)  Cf  particular  importance  to  the  project  manager  in  the  acquisition 
process  are  tne  following  policy  statements: 

a.  Decisions  to  acquire  AD?D  will  be  preceded  by  and  predicated 
upon  the  results  of  well-documented  studies  that  indicates: 

1.  That  a valid  information  requirement  exists.  The  functions 
or  processes  to  be  acccmplisned  through  tne  use  of  automatic  processing  are 
essential  to  mission  requirements. 

2.  That  automatic  processing  is  the  most  cost-effective  means 
of  satisfying  tne  requirements. 

5*  That  tne  system  to  be  employed  has  been  designed  to  acnieve 
tne  nignest  practical  degree  of  effectiveness  and  operational  economy. 


8 


T 


4.  That  tne  lowest  overall  cost  alternative  for  satisfying 
tne  requirements  nas  bean  determined  prior  to  selection  and  acquisition  of 
ADPc.  resources. 

b.  Specifications  to  support  tne  acquisition  of  ADPn  resources 
will  be  developed  independently  of  a specific  vendor's  products,  Equal 
opportunity  and  consideration  will  bo  accorded  to  all  vendors  who  offer 
products  capable  of  meeting  the  specifications. 

c.  The  method  of  acquisition  will  offer  the  greatest  advantage 
to  the  Government  under  tne  circumstances  surrounding  the  situation. 

d.  To  further  promote  effective  selection  and  acquisition  of  AEPE 
resources,  a professionally  staffed  activity  witn  primary  full-time  mission 
to  develop  solicitation  documents,  evaluate  vendor  responses  and  competi- 
tively select  ADPa  will  be  establisned  within  e3cn  military  department. 

In  addition  to  tne  technical  and  management  policies  contained  in  tne 
directive,  selection  and  acquisition  of  A D?i  resources  will  oe  in  accord- 
ance with  tne  policies  and  procedures  of  the  Federal  Property  Management 
Regulation  and  the  Armed  Services  Procurement  Regulations  (ASPR). 

In  tne  event  of  conflict  between  tne  two  government  regulations,  the  pro- 
visions of  the  FPKR  govern. 

The  directive  also  delegates  responsibilities  to  the  secretaries  of 
tha  military  services  to  approve  the  selection  of  ADPi  resources  and  to 
issue  appropriate  implementing  documents  and  procedures. 
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2.3  Past  and  Present  Methodology  in  System  Procurement 


In  the  early  days,  the  existence  of  an  operational  computer  system 
was  virtually  an  end  in  itself.  Different  systems  were  compared  with 
easily  observable  quantitative  characteristics  such  as  memory  size,  number 
of  instructions  executable  in  a second,  speed  of  printers,  card  readers,  or 
clock  rates  of  the  equipment.  As  the  logical  design  and  usage  of  informa- 
tion processing  equipment  became  more  complex,  it  became  apparent  tnat  such 
easily  observed  or  measured  physical  parameters  did  not  always  yeild  an  in- 
ference as  to  "quality"  or  "goodness"  which  correlated  well  with  ones  intu- 
itive feeling  about  the  relative  work  or  usefulness  of  differeny  systems. 
(22,  2)  In  tne  19?C's,  independent  sub-systems  were  designed  for  inter- 
dependent activities.  Further,  the  systems  of  tne  195C'8  were  largely 
operational-level  systems . Tney  provided  the  information  needed  by  first- 
level  supervisors  and  tneir  subordinates.  (6,  196)  From  this,  it  is  evi- 
dent that  tne  early  6/steas  were  indeed  easier  to  evaluate.  Concreteness 
and  simplicity  of  design  as  well  as  ease  of  application  were  the  major  con- 
tributing factors.  As  the  systems  grew  in  scope  and  complexity,  so  did  the 
problems  of  evaluation  and  selection.  The  independent  sub-system,  such  as 
the  payroll  programs,  oecame  a small  part  of  a larger  financial  system. 

*ith  system  complexity  came  evaluation  complexity  where  systems  analy- 
sis techniques  lagged  one  generation  behind  those  machines  which  were  to  be 
acquired.  Cnly  in  recent  years  have  techniques  narrowed  the  gap  between 
the  hardware  and  its  evaluation.  The  stress  in  performance  evaluation  has 
snifted  from  programming  and  testing  of  new  systems  to  the  "front  end"  ap- 
proach waich  includes  documentation  of  tne  present  system,  evaluation  of 


system  requirements,  and  designing  the  improved  system.  (6,  16?)  As  the 
■tress  on  analysis  grew — directly  affected  by  tne  systems  coaplexi ty--so 


grew  tne  mstnodology  available  to  t:.e  analyst.  Tne  period  between  1°20 
and  19^C  saw  tne  development  of  paperwork  and  process  flowcharts.  Their 
major  shortcoming's  were  tne  lack  of  the  identification  of  data  elements  and 
. volumes.  Curing  the  195C's,  general  flowcharts  and  block  diagrams  evolved 

_ based  on  previous  attempts  at  accuracy.  The  period  196C-197C  saw  the 

greatest  progress  in  the  analysis  of  computer  systems.  NCR's  Accurately 
Defined  System  (ADS)  was  an  improvement  on  the  use  of  charts  since  it  pro- 
vided a well-organized  and  correlated  approach  to  system  definition  and 
specification.  (6,  173) 

ACS  used  five  interrelated  forms  to  provide  tne  system  (application) 
definition.  Tne  process  began  with  the  definition  of  output.  Next,  in- 
puts were  defined  — on  tne  second  fcrm.  Tne  third  form  provided  the  defini- 
tion of  computations  to  be  performed  and  tne  rules  of  logic  governing  the 
computation.  Tne  interrelationship  of  computations  were  also  defined  on 
this  form,  as  were  tne  sources  of  information  used  in  tne  computation.  The 
fourtn  form,  the  history  definition,  specified  information  to  be  retained 
beyond  tne  processing  cycle  for  subsequent  use.  The  fifth  form  provided 
*‘  the  logic  definitions,  ir.  the  form  of  a decision  table. 

Within  ADC,  .Information  linkage  was  accomplisned  in  two  ways.  First, 
each  data  element  was  assigned  a specific  tag  or  reference.  Next,  each 
time  tne  tag  was  used  in  tne  system,  it  was  linked  back  to  tne  previous 
link  in  tne  chain.  All  elements  of  data  were  chained  from  input  to  output, 
accomplisned  tnreurn  tne  use  of  p-ge  and  line  numbers,  lhe  process  of 
chaining  facilitated  identification  of  omissions  and  contradictions  in  the 
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*ys tarn.  Once  tne  information  requirements  were  .establisned,  the  system 
design  pnase  determined  the  appropriate  hardware  mix  to  effect  the  system. 

Anotner  approacn,  .cnown  as  information  algebra,  based  or*  tue  efforts 
of  Mr.  Robert  dosak,  provided  tne  tneory  for  systems  specification.  Infor- 
mation algebra  was  an  important  development  because  it  provided  a theoreti- 
cal basis  for  automatic  processing  of  system  specifications.  The  primary 
intent  of  information  algebra  is  to  extend  tne  concept  of  stating  the  rela- 
tionships among  data  to  all  aspects  of  data  processing.  This  will  require 

tne  introduction  of  increased  capability  into  compilers  for  translating 

« 

this  type  of  relational  expression  into  procedural  terms. 

while  both  of  tne  foregoing  wers  Da  sad  on  tne  assumption  tnat  tne 
study  of  tne  organization  and  its  needs  nad  been  completed,  two  new  develop- 
ments, ARDI  (Analysis,  Requirement  Development,  Design  and  Development, 

Implementation  and  evaluation)  developed  by  Pnilips,  and  Study  Organization 
Plan  (sCP)  developed  by  liM,  aided  tne  initial  phase  of  analysis.  The  lat- 
ter is  tne  more  significant  contrioution  to  tne  field  because  if  pulled 
various  teciiniques  togetner  into  an  integrated  approach.  SOP  was  designed 
to  gather  data  witn  which  to  analyze  the  information  needs  of  tne  entire 
organization.  (6,  17^*) 

Another  improvement  which  made  analysis  easier  was  the  use  of  the 
ti06lcyns  System.  (6,  191)  'Jsing  tne  rfoskyns  approach  the  system  is  de- 
scribed in  terms  of  programs  and  files  witn  tne  programs  being  described 
in  terms  of  records  and  data  elements.  These  sets  of  relationsnips  sre  re- 
corded in  tne  form  of  matrices.  In  summary,  tne  Hoskyns  system  accepts 

i 

system  specifications  and  converts  tnec  to  C030L  programs  without  manual 
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intervention.  Tne  system  was  developed  and  implemented  in  three  British 
corporations  by  hos^yns  Systems  Research  Incorporated.  It  was  introduced 
in  tne  United  States  in  197^  and  is  in  use  at  Xerox,  General  Foods,  and 
Allied  Chemical  . 

•inereas  tne  first  and  second  generation  analysis  techniques  concen- 
trated on  suooptimization  within  a given  organization,  third  generation 
■ystens  pnilosopny  is  concentrating  on  studying  the  organization  as  a 
whole.  This  new  idea  presented  an  even  larger  problem  for  tne  analyst 
until  a new  approach  wae  devised.  This  is  known  as  Problem  Stated  Lanruage/ 
Problem  Stated  Analyzer  (PSL/PSA).  The  concept  wsb  now  different.  The 
analyst  asked  what  he  wanted  to  inspect  regardless  of  how  those  needs 
snould  be  met.  The  Problem  Statement  Language  (PSL)  is  designed  to  express 
desired  system  outputs,  tne  data  elements  whicn  comprise  these  outputs,  and 
formulas  to  compute  tneir  values.  Tne  user  specifies  the  parameters  which 
determine  tne  volume  of  inputs,  and  tne  outputs  end  the  conditions  (partic- 
ularly those  related  to  time)  wnicn  govern  tne  production  cf  outputs  and 
the  acceptance  of  inputs.  The  Proolem  statement  Analyzer  (PiA)  accepts  in- 
puts in  PaL  and  analyzes  tnem  for  correct  syntax,  them 

a.  Produces  compremnsive  data  snd  function  dictionaries. 

b.  Perforn.3  static  network  analysis  to  insure  completeness  of 
derived  relationsnips . 

c.  Performs  dynamic  analysis  to  indicate  time-dependent  relation- 
ships cf  data . 

d.  Analyzes  volume  specifications. 

bo  far  only  tr.e  analysis  aspect  of  computer  systems  design  and 
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•valuation  techniques  have  been  discussed.  The  question  still  to  be  an- 
swered is:  lnhat  methodology  is  available  for  testing  the  newly  designed 

sy stoat 

As  previously  stated,  the  earlier  systecs  were  relatively  easy  to 
evaluate  due  to  tnsir  similar  cna racteristics  of  3ize,  number  of  instruc- 
tions executable  in  e given  period  of  time,  etc.  As  the  configurations  be- 
caae  more  complex  and  multiprogramming  and  multiprocessing  became  more 

prominent,  the  evaluation  techniques  available  had  to  undergo  complete 

« 

changes  in  order  to  meet  these  new  innovations. 

The  earliest  attempts  at  C?’J  evaluation  were  the  two  interdependent 
“instruction  mixes"  and  "kernel*  methods.  In  the  mix  metncd,  each  instruc- 
tion or  related  group  of  instructions  in  the  repertoire  of  a computer  is 
assigned  a weighting  factor  obtained  by  analysis  or  measurement  of  a pro- 
gram or  programs  in  execution.  Applying  the  weirnt  to  eacn  instruction 
provides  an  average  instruction  time  t.iat  can  form  a basis  of  comparison 
between  two  or  more  systems.  (9,  257)  Tne  technique  in  tne  kernel  method 
is  to  determine  tr.e  most  frequently  used  portions  of  an  application  and  to 
program  tnese  portions  in  tne  various  instruction  sets  of  the  cnetral  pro- 
cessing units  being  compared,  /tfter  eacn  kernel  nad  been  evaluated,  they 
were  combined  according  to  6cme  weighting  function.  Although  tne  kernel 
metnod  was  a better  tool  tnan  tne  instruction  mix,  both  methods  nave  now 
been  discarded  in  most  evaluation  and  selection  procedures.  The  instruc- 
tion mix  is  no  longer  used  since  a single  weirnt  is  used  in  evaluating  the 
performance  of  systems  with  different  instrjction  sets,  memory  configura- 
tion, etc.  (°,  257)  The  kernel  method  has  fallen  into  disuse  primarily 
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because  it  ignores  input/output  considerations  and  software  performance 
factors.  Moreover,  it  is  usually  difficult  or  impossible  to  relate  the 
time  for  an  assortment  of  kernels  to  a given  user's  real  time  data  pro- 
cessing applications.  (22,  6) 

sith  tne  instruction  mix  and  the  kernel  method  falling  into  disuse, 
the  bencnmark  method  gained  in  popularity.  A commonly  accepted  definition 
of  the  bencnmark  metncd  is  a program  or  programs  which  seek  to  represent  by 
representative  programs  the  total  automatic  data  processing  (AD?)  workload. 
(5,  4l)  In  the  context  of  this  definition,  the  current  workload  with  ex- 
pansion is  designed  and  executed  on  several  ccnfi p-ure tions . The  results 
are  then  compared.  Although  quite  popular,  the  method  has  some  inherent  or 
potential  pitfalls.  The  definition  of  a "representative''  workload  is  most 
probably  tne  most  prominent.  The  central  difficulty  lies  with  establis.oing 
the  representativeness  of  the  Job  or  Jobs  being  run  considering  that  they 
are  usually  a very  small  sample  of  tne  actual  workload  planned  for  tne  sys- 
tem in  question.  (22,  7) 

Otner  factors  whicn  are  known  to  affect  tne  outcome  of  any  benchmark 
test  are  "CPU  utilization,"  "channel  utilization,"  "terminal  response  time," 
etc.  To  aid  in  tne  evaluation  of  these  factors,  simulation  has  become  quite 
popular  due  to  its  flexibility,  despite  the  high  cost  and  time  factors  in- 
volved in  writing  and  running  tne so  packages.  During  the  196c's,  several 
simulation  languages  were  developed  and  now  operate  with  a high  decree  of 
proficiency. 

Finally,  hardware  and  software  packages  were  end  are  still  being  devel- 
oped to  aid  t.»e  performance  evaluator.  3otn  packages  have  limitations, 
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but  they  are  acceptable  as  tools  which  help  in  the  validation  and  verifica- 
tion of  programs  being  executed  on  a system.  The  software  monitor  is  a 
program  which  is  usually  imbedded  amongst  tne  opera tinr  system  and  produc- 
tion programs  of  a system.  As  the  system  executes,  data  is  collected  which 
states  something  about  the  sequence  of  events,  queues,  timings,  memory 
cycles,  etc.  Tne  drawback  to  these  software  packages  is  the  fact  tnat  the 
measuring  program  itself  can  adversely  affect  the  system  being  measured. 

The  measuring  program  is  using  the  system’s  memory,  input/output  devices, 

( 

etc. — exactly  those  components  wnicn  it  is  to  measure.  To  overcome  this 
problem,  hardware  monitors  have  been  developed  which  are  connected  to  key 
points  in  tne  system.  Although  tney  do  not  interfere  with  the  operation  of 
the  system,  their  drawback  lies  in  tne  limited  number  of  system  points  that 
can  be  connected  to  the  monitor. 

The  cost  of  system  development  and  procurement  are  not  cheap.  i*jiny 
managers  enter  into  tne  arena  without  any  idea  of  the  cost  distribution  in- 
volved. Figure  2-1  presents  an  overview  of  tne  systems  development  costs  as 
they  existed  in  the  recent  past.  It  is  worthy  of  note  that  more  effort  is 
required  during  tne  initial  phases  of  tne  computer  project.  Both  tr.e  amount 
of  cost  and  the  distribution  of  resources  have  changed.  In  first  generation 
systems,  Phases  I and  II  absorbed  approximately  five  percent  of  system  de- 
velopment cost.  The  expanded  scope  and  sophistication  of  third  generation 
systems  has  increased  overall  development  cost,  with  approximately  twenty 
percent  absorbed  by  Phases  I and  II. 
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Figure  2-1 


2.4  Sola  Source  vs.  Competitive  Bids 


Today  most  computer  acquisitions  are  made  tnrough  competitive  bids, 
tnat  is,  more  t.nan  one  vendor  responds  to  tne  Request  for  Proposal  (RF?) 
and  offers  wnat  he  believes  to  be  the  best  system  or  sys*'ms.  Procurement 
of  AD?  is  generally  dons  on  tne  basis  of  tnree  general  principles.  It  may 
be  obtained  by  general  performance  specifications.  Tnis  normally  applies 
to  an  entire  system.  Second,  the  user  may  only  have  need  for  an  equipment 
specification  (i.e.,  terminal)  in  wnich  case  nis  requirements  are  stated  in 
terms  of  a piece  of  equipment.  Lastly,  specified  ADP  may  be  requested 
(i.e.,  13a.  36c)  by  tne  user.  In  essence,  specified  ADP  is  a make  and  model 
description.  (1,  4)  The  federal  government  generally  requires  competitive 
bids  on  any  major  system.  Few  exemptions  are  granted,  and  only  then,  if  it 
can  be  s.nown  that  sole  source  acquisition  is  the  most  cost  effective,  llost 
sole  source  acquisitions  are  justified  under  several  circumstances.  As  an 
example,  only  one  vendor  has  the  specific  equipment  needed  or  the  conver- 
sion effort  is  too  great.  Tne  key  ingredient  in  tne  request  for  specified 
ADPn  or  sole  source  is  tne  rationale  to  support  how  the  requested  aDPo  was 
evaluated  and  w.ny  it  was  determined  to  be  the  best  or  only  equipment  whicn 
will  satisfy  the  requirement  or  need.  (27,  51)  Sole  source  has  its  ad- 
vantage. The  evaluation  becomes  a routine  task  in  tnat  tne  user  makes  a 
rat ner  perfunctory  cneck  of  system  capability.  In  this  respect,  the  eval- 
uation and  selection  method  costs  very  little.  Sols  source  acquisition 
rests,  of  course,  on  tne  premise  tnat  the  user  knows  tne  equipment1 s capa- 
bility and  his  own  needs  well  enourn  to  be  able  to  pick  a specific  manu- 
fa  cturer . 
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Tnis  type  of  acquisition,  however,  does  a disservice  to  most  users. 

No  natter  what  the  requirements  say  be,  there)  are  at  least  two  vendors  who 
will  oe  able  to  meet  then.  If  tnis  is  not  possible,  tnen  it  nay  be  advis- 
able to  look  at  tne  overall  operation  ana  see  if  changes  cannot  be  institu- 
ted which  will  allow  greater  latitude.  Competitive  bids  allow  tne  user  to 
pick  and  choose  his  equipment  based  cn  need,  cost,  and  performance.  In  some 
cases,  vendors  have  bid  systems  wnose  configurations  were  far  superior  to 
tnose  that  the  user  had  in. mind.  There  are  disadvantages  to  tnis  method  of 
procurement.  Competitive  bids  are  costlier  than  sole  source  since  the  ex- 
isting system  must  first  be  documented;  second,  an  RF?  must  be  prepared; 
third,  tne  validation  of  tne  proposal  including  system  performance  measure- 
ment must  be  made;  snd  lastly,  the  actual  system  must  be  selected. 


/ 
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2.5  Evaluation  and  Selection  by  permanent  Staff  versus  Consultant  Firm  or 


Ad  Hoc  Committee  'Jsare 

For  those  organize tions  whose  primary  mission  demands  the  support  of 
one  cr  sore  nD?  facilities,  tne  need  for  a permanent  staff  of  highly  trained 
computer  professionals  nas  grown  from  one  of  novelty  to  one  of  necessity. 
Tecnnological  advances  have  coze  witn  startling  rapidity  as  coaputing  ma- 
c nine 3 spread  over  a spectrua  ranging  froa  tne  zinicozputers  to  tne  "number 
cruncners."  These  individuals  are  usually  up-to-date  in  tneir  speciality 
of  hardware  configurations,  software,  simulation,  etc.  They  are  familiar 
witn  tne  organization's  problems  and  tney  play  an  active  and  vital  role  in 
tne  evaluation  and  selection  process.  In  those  areas  which  are  sensitive 
because  of  security  or  proprietary  matters,  a permanent  staff  is  usually  a 
must.  Despite  these  advantages,  a permanent  staff  has  one  real  and  one 
potential  drawbac«c--cost  and  tunnel  vision  or  stagnation.  A professional 
group  is  quite  expensive  and  adds  to  tne  corporate  overnead . As  much  as 
eighty-five  psreer. t of  a facility's  total  budget  has  been  assigned  to  the 
maintenance  of  programming  and  analysis  support.  Tunoel  vision  is,  of 
course,  only  a potential  problem.  As  the  group  becomes  familiar  with  the 
current  equipment  in  its  day-to-day  activities,  tne  equipment  itself  will 
impose  its  limitations  and  constraints  on  those  directly  dealing  witn  it. 

In  tne  future,  suggestions  for  solving  problems  will  be  rejected  because 
tne  limitations  cf  tne  current  system  do  not  allow  tne  suggested  solution. 
Tney  will  normally  not  be  rejected  because  they  are  unworkable. 

The  use  of  consultants  must  be  considered.  Ahilo  the  consulting  firm 
is  less  expensive  tnan  tne  maintenance  of  a permanent  professional  staff, 
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it  contains  all  the  merits  of  the  latter.  The  firm  is  employed  only  for 
tr.e  development  tnrough  the  test  and  evaluation  phases  of  the  system  and 
usually  ends  its  association  with  tne  organization  after  the  last  piece  of 
equipment  has  been  installed  and  accepted.  As  an  example  the  U.  b.  Army's 
supply  system  in  the  pacific  area  of  operations  (5S)  was  developed  and  main- 
tained by  tne  Computer  Services  Corporation  (CSC).  In  f&ct,  CbC  provided 
systems  personnel  on-site  in  Vietnam  to  maintain  the  system.  In  some 
instances  tne  firm  will  be  retained  on  a part-time  basic — ^ust  in  case  prob- 
lems  develop  after  tne  system  is  installed.  Again,  some  typical  pitfalls 
must  be  identified.  If  tne  consultant  has  enjoyed  great  success  witn  a 
particular  application,  he  may  recommend  a repeat  performance  with  en  i- 
dentical  cr  similar  system  even  thougr.  it  may  net  quite  meet  the  organiza- 
tion's needs.  Depending  on  the  consultant's  own  background,  experience, 
and  inclination,  an  organization  acquiring  new  and  unfamiliar  equipment  may 
be  persuaded  to  purc.nase  a more  expensive  system  from  a particular  vendor 
since  tnese  products  are  most  familiar  to  the  consultant. 

In  most  cases,  the  ad  noc  committee  is  the  least  expensive  path  to 
follow  in  choosing  an  evaluation  and  selection  team.  Members  of  the  team 
are  normally  from  tae  organization  acquiring  the  computer,  therefore,  they 
are  usually  familiar  with  the  problems  and  as  a result  little  time  must  be 
wasted  in  explaining  the  processing,  needs,  and  policy  of  the  organization. 
Mr.  h.  M.  Timmreck  suggests  tnat  this  is  quite  satisfactory  for  acquiring 
an  upgraded  system.  (26,  2CC)  They  will  essentially  edd  a few  AD?  compo- 
nents to  the  system,  make  tne  necessary  comparisons,  and  purchase  the  cheap- 
est system.  This  method  is  also  equally  workable  if  a sole  source 
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procurement  is  desired.  The  drawbacks  to  the  ad  hoc  committee  are  obvious. 
The  group  nas  little  expertise  in  computing  machinery  in  general  and  no 
experience  at  all  in  computer  evaluation.  If  the  co.ntarct  has  a high  dol- 
lar value  and  cany  vendors  are  invited  to  bid,  it  is  safe  to  assume  that 
tne  aystec  wnicn  is  finally  purcnased  is  not  tne  best  system  that  could 
have  been  acquired  for  the  coney  expended. 

What  then  is  tne  proper  combination?  Although  no  firm  rules  are  avail- 
able, Mr.  cdward  C.  Joslin . suggests  all  tnree  cay  be  used  depending  upon 
the  expected  system's  complexity,  the  ad  hoc  committee's  understanding  of 
tne  computer  evaluation  and  selection  process,  and  the  experience  of  the 
permanent  staff.  In  the  majority  of  cases,  a combination  of  all  tnree 
groups  is  used.  Members  of  tne  ad  hoc  committee  are  usually  appointed  pro- 
ject managers  witr.  the  consultants  and  tne  permanent  staff  of  AD?  profes- 
sionals assigned  supporting  functions  suc.n  as  costing,  bencnmark  definition, 
etc* 
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2.6  Vendor  versus  User  Considerations  in  System  Procurement 


Although  the  title  of  this  section  nay  imply  that  an  antagonistic  re- 
la  tionaaip  exists  between  any  given  vendor  and  user,  little  evidence  appears 
to  support  this  view.  One  fact,  however,  is  obv.  us.  The  vendor  wants  to 
sell  his  line  of  equipment  and  he  hopes  to  persuade  the  potential  user  to 
purchase  it. 

Assume  for  tne  moment  that  the  RF?  has  been  completed  stating  tne  man- 
datory requirements  end  the  desirable  features.  The  organization  expects  a 

i 

large  response,  yet  it  receives  only  a few  politely  worded  inquiries.  *nat 
are  the  possible  causes: 

a.  The  RF?  addresses  few  vendors  who  manufacture  a mandatory 
piece  of  equipment. 

b.  To  bid  is  too  costly  for  a large  number  of  vendors.  This  is 
one  point  that  seems  to  have  been  overlooked  eitner  due  to  unawareness  by 
those  w.no  write  tne  RF?  or  due  to  limited  resources  whicn  will  be  expended 
on  tne  purcnase.  Ridding  can  be  expensive,  preparing  a reply  to  a propos- 
al can  cost  anywnere  from  jl,CCC.CC  to  well  over  *ACC,CCC.CC.  This  expense 
must  be  borne  by  the  vendor  who  nas  no  guarantee  tr.at  he  will  be  awarded  tne 
contract.  For  instance,  at  JTJlVAC's  Marketing  Test  Center  usually  two  to 
five  bencnmarks  are  in  process  with  another  ten  to  fifteen  in  various  stages 
of  completion.  An  sverage  bencnmark  takes  from  six  to  twelve  weeks  to  com- 
plete. This  process  uses  approximately  ICC  hours  cf  actual  computer  time 
and  from  sixty  to  seventy-five  total  manweeks.  Seme  calculations  can  quick- 
ly verify  the  fact' that- an  Rr?,  describing  any  system,  is  an  expensive  pro- 


position for  any  vendor 


c.  Tne  RFP  is  not  clearly  written.  The  RFP  should  be  precise  in 
its  wording,  leaving  nothing  to  tne  fruitful  imagination  or  to  the  "benefit- 
myself*  interpretation  of  tne  vendors.  Unless  sole  source  acquisition  can 
be  justified,  tne  request  should  be  general  enougn  in  its  requirements  to 
allow  as  many  vendors  as  possible  to  complete  witnout  comprimising  the 
needs  of  tne  organization,  furtner,  a well-written  RTF  steers  the  vendors 
in  the  right  direction.  They  need  not  waste  their  resources  in  trying  to 
guess  what  the  buyer  really  wants. 
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CHAPTER  III 


Trim  SaLiCTICN  RRCCiriS 

J .1  Analysis  and  Specifications  of  tnu  Present  system 

As  was  snown  in  Section  2.p,  aora  and  core  tiae  and  effort  is  expended 
at  tne  *front-end,*  tiiat  is,  during  the  initial  phases  of  any  computer  ac- 
quisition. depending  on  the  author,  any  number  of  steps  can  be  cited  which 
should  be  included  in  this. initial  period.  Mr.  Joslir,  one  of  the  most 

t 

prolific  writers  on  tnis  subject,  suggests  three  basic  steps  which  should 
be  followed  in  the  beginning  efforts.  The  three  steps  ares  (1)  data  gath- 
ering or  investigation  of  the  present  system  plus  any  new  requirements; 

(2)  analysis  of  tne  data  gathered  in  the  investigation;  and  (5)  syntnesis, 
or  refitting  of  the  parts  and  relationsnips  uncovered  through  analysis  into 
a better  system,  (ljf,  6p) 

One  important  point  must  be  discussed  before  this  topic  can  be  furtner 
explored.  Is  tnis  acquisition  tne  user's  first  or  does  ne  already  have  a 
system?  Mr.  limmreck  joints  out  that  drawing  up  specifications  of  need  for 
tne  former  will  be  much  more  difficult.  (2c,  2C5)  For  example,  the  small 
organization  without  a computer  must  first  very  carefully  ask  itself  wheth- 
er it  really  needs  one.  It  is  quite  possible  that  the  company's  needs  can 
be  satisfied  by  other  means  ratner  tnan  purchase.  The  fallacy  of  owninr  a 
system,  and  one  that  may  be  too  large,  was  shown  in  the  early  part  of  the 
197C's.  Over  2CC  software  houses  and  service  bureaus  closed  their  doors. 
Overspending  on  computer  systems  coupled  with  u declininr  need  and  a slow- 
ing economy  were  the  chief  reasons  for  these  closures. 
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In  either  case,  a very  thorough  examination  of  tne  organizational  ob- 
jectives snould  be  'undertaken.  Since  members  of  the  teas  cone  and  go, 
tnese  objectives  should  be  in  writing  and  given  to  each  member  of  the  tear. 
(l}f,  65)  These  objectives  should  be  specific  enough  so  that  a lonr-range 
plan — 5 years  or  more — can  be  developed.  A thorougn  examination  of  tne 
organization  must  oe  made.  3 y tnis  I mean  "who  dees  what  to  whoa"  and  wnat 
are  tneir  interrelationships,  historical  data  must  also  be  gathered,  why 
is  the  present  system  operating  in  this  manner?  ..hat  are  the  policies, 
practices,  and  regulations?  The  analysis  should  focus  on  detailed  break- 
downs of  the  workload  classes,  isolate  specific  projects,  organizations, 
and  personnel  requesting  service.  Statistics  showing  tne  total  machine 
time  used  and  the  number  of  jobs  each  of  the  aforementioned  ran  in  a given 
period  must  be  gathered.  Priorities  and  their  assignment  procedures  must  be 
analyzed.  Many  organiza tions  have  some  dedicated  jobs  which  must  be  exe- 
cuted at  specific  intervals,  i.e.,  payroll.  These,  too,  must  be  isolated 
and  tneir  time  and  number  recorded.  The  type,  duration,  and  priority  of 
the  backlog  must  be  examined.  The  means  of  backlog  resolution  must  be 
known.  Furtner,  tr.o  idle  time  of  the  system  must  be  determined.  Specifi- 
cally, this  pnase  snould  be  divided  into  operational  and  system  characteris- 
ti  cs . 

Operational  characteristics  fall  into  several  categories.  Questions 
concerning  common  operator  errors  whicn  affect  thruput,  job  scheduling,  user 
dialogue,  document  problems,  availability  of  tapes  and  discs,  etc.,  are 
certainly  germane.  Users  of  tne  system  should  be  questioned  concerning 
turnaround  adequacy.  The  system  logs  should  be  examined  to  determine  the 
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number  of  hours  par  recording  period  in  whicn  tne  machine  was  emulating, 
simulating,  idle,  compiling  or  assembling,  inoperable  due  tc  hardware  cr 
software  failure,  or  down  due  to  preventive  maintenance . Dome  statistics 
should  be  gathered  on  tne  number  of  forms  wnic.n  are  used.  Finally,  re- 
sources expended  in  both  man-hours  and  machine  time  must  be  examined. 

The  system's  characteristics  must  also  be  examined.  Data  concerning 

CPU  utilization  per  montn,  tne  peaks  and  valleys  of  the  jobs  in  core,  the 

number  of  input/output  (I/C)  operations,  the  support  equipment  available 

( 

(such  as  tapes  and  discs),  opinions  from  systems  programmers  on  the  limit- 
ing factors  (core,  disc,  memory,  channels,  etc.)  must  b9  obtained.  If  pos- 
sible, information  on  the  time  spent  for  solving  production  problems,  aid- 
ing users,  maintaining  old  applications,  developing  new  programs,  and  alter- 
ing software  must  be  gathered. 

Tne  next  step  is  to  consider  tne  areas  of  process  descriptions  and 
narrative  flowcharting.  Process  description  concerns  inputs  and  outputs. 

It  essentially  involves  interviewing  those  individuals  who  receive  reports 
or  those  wno  generate  tnem.  Areas  of  investigation  are  tne  report  need, 
format,  and  frequency.  The  purpose  of  the  investigation  is  tnreefold: 

(1)  determine  exactly  what  is  used;  (2)  what  can  be  eliminated;  and  (5) 
what  else  is  needed.  It  is  nere  that  each  job  or  group  of  jobs  must  be 
thoroughly  investigated.  Cnaracteri sties  of  each  input/output  should  show 
CPU  usage  per  job  in  both  prime  time  and  non-prime  time.  The  analysis  must 
snow  for  each  job  tne  number  of  tapes  and  discs  used,  the  percent  that  com- 
pile or  assemble,  tne  language  eacn  one  uses,  the  percent  cf  CPU  time  used 
(for  production  jobs),  tne  number  of  production  jobs,  and  I/C  times.  As  is 
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tne  case  in  cost  facilities,  there  are  always  a few  programs  whicn  are 
classified  as  tne  largest,  the  most  executed,  and  tne  most  important.  It 
is  advisable  to  isolate  tnese  runs  and  prepare  separate  information  sneets 
cn  eacn.  Kucn  of  tne  present  system  may  be  shortened,  rewritten,  or  total- 
ly eliminated.  Inis  is  particularly  true  for  a very  large  system  that  has 
been  in  existence  for  a number  of  years.  Tne  tendency  is  to  expand,  unfor- 
tunately not  always  in  an  orderly  fasnion.  Usually  much  redundant  informa- 
tion is  gatnered.  Depending  on  the  accuracy  cf  the  reporting  system,  cur- 
rent evaluation  criteria  and  reports  may  be  used.  The  last  portion  of  this 
subject  concerns  processing  or  in  other  words  converting  given  inputs  into 
desired  outputs.  In  tnis  connection  one  must  determine  the  file  sizes,  up- 
date frequencies  and  number  of  records  per  file,  plus  a host  of  other 
varia  oles . 

Narrative  flowcharting  is  tne  process  of  documenting  the  present  sys- 
tem. It  shows  tne  current  processes  and  subprocesses  at  various  stages  of 
completion.  Narratives  sneuid  be  as  specific  as  possible  when  dealing  with 
rates,  volume,  standards,  and  pealcloads  since  tnese  will  be  used  in  work- 
load determination  and  benenma r<cs . It  is  the  purpose  of  the  narrative  to 
cross-reference  all  tne  I/C's  wnicn  were  gathered  in  tne  process  descrip- 
tion . 

The  second  step  in  this  overall  process  is  the  analysis  of  the  infor- 
mation whicn  has  been  gatnered.  All  I/C's  should  now  be  logically  ar- 
ranged in  sequence.  Tne  pertinent  files  are  examined.  The  relationships 
between  files  must  again  be  questioned.  Dome  files  may  be  dropped,  modi- 
fied, or  carried  as  they  are  presently  designed.  The  information  rathered 


25 


up  to  tnis  point  snould  now  give  a clear  picture  of  tne  various  functions 


whicn  are  directly  related  or  affected  by  tne  current  system.  Tne  data  can 
be  utilized  fcrs 

a.  Job  accounting — C?'J  time  ♦ tape,  etc. 

b.  Pro-rag  analysis — number  of  test  runs  and  their  frequency; 
machine  time  required  for  test  runs;  peripheral  resources  used. 

c.  Kultiprorramminr  effectiveness — machine  hours  overlapped,  num- 
ber of  tasks  concurrently  processed,  CP'J  usage,  I/O  usage,  idle  time. 

« 

d.  Opera tior.3  analysis — idle  time,  set  up  time,  efficiency  of 

schedule. 

e.  Program  profile — processing  requirements,  I/O  dependency  of 
lobs,  CP'J-l/C  balance. 

f.  Resource  utilization — resources  used  by  load  modules,  load 
module  frequency  utilization,  l/C  dependency  of  loud  modules. 

g.  Hardware  analysis — core  size  impact,  CPU  speed  impact,  I/O 
device  impact,  channel  impact. 

At  tne  conclusion  of  this  phase,  tne  entire  system  as  it  currently 
exists  should  be  known  in  the  fullest  detail.  The  analysis  may  bring  seme 
unexpected  surprises  in  tnat  tne  findings  may  show  tnat  a new  system  is  not 
needed.  Dropping  of  redundant  information  or  tne  acquisition  of  more  memo- 
ry, a disc,  or  several  tapes  may  be  all  that  is  needed.  Although  only  a 
few  such  cases  can  be  found  in  the  literature  many  more  are  likely  to  exist. 
(25,  1225-125J) 

The  final  step  is  tnat  of  synthesizing  this  information  with  those 
objectives  tnat  must  be  met  by  tne  new  system.  Five  factors  may  be 
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considered  impc.-tant  in  a good  system  design: 

a.  Try  to  minimize  input  and  output  operations. 

b.  »nere  possible,  source  inferm-tien  saould  be  initially  trans- 
ferred directly  into  a media  acceptable  as  input  to  the  computer. 

c.  beek  multiple  uses  of  common  source  data. 

d.  Attempt  to  keep  tne  system  simple,  flexible,  reliable,  econom- 
ical, and  acceptable  to  the  users. 

Again,  as  this  evolution  progresses,  steps  must  be  taken  to  rid  tne 
system  of  many  anachronisms  which  crept  in  over  time,  and  were  later  de- 
clared law.  The  word  here  is  standardization — a costly,  theum  worthwhile, 
effort.  Mr.  Timmreck  states,  "Clearly,  an  organize tion' e approach  to 
standardization  will  si cnificantly  affect  its  flexibility  when  selecting  a 
computer  system.  An  organization  wnich  strongly  emphasizes  programming 
standards  will  normally  be  relatively  free  to  switch  from  one  manufacturer 
to  another."  (26,  2C4)  «ith  government  support  or  pressure,  mere  and  more 
aspects  of  tne  computer  industry  are  beinr  standardized . Tne  federal  gov- 
ernment's insistence  on  tne  usage  of  C03CL,  as  well  as  tne  current  attempts 
by  tne  Army,  Navy,  and  tne  National  Bureau  of  Standards  to  arrive  at  stand- 
ard benenmarks  serve  as  examples  of  standardization. 

As  all  of  the  information  is  synthesized,  various  computer  configura- 
tions must  be  studied.  Flowcharts  which  indicate  volumes,  record  length, 
I/C  device  type,  frequency  of  execution,  etc.,  should  be  used.  At  this 
stage,  trade-offs  must  be  made.  As  an  example,  discs  may  be  a favored 
media;  however,  the  application  may  be  done  on  tape  _Just  as  well.  In  this 
esse  the  price  of  tne  devices  may  be  tne  determining  factor.  The  most  sat- 
isfactory metned  of  determining  a system  design  is  to  develop  from  "the 
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system*  downward  listing  as  many  variations  as  meet  the  need, 
has  been  completed,  it  is  possible  to  choose  parts  of  each  in  order  to 
arrive  at  tne  optimum  system  which  meets  all  constraints.  In  essence,  a 
nybrid  system  is  developed.  After  tne  final  system  is  designed,  it  should 
serve  as  e basis  for  comparison.  It  should  not,  nowever,  be  used  as  an 
absolute  system,  macn  vendor  will,  no  doubt,  oid  one  or  mere  systems  which 
may  be  radically  different  from  the  synthesized  configuration,  yet  meet  the 
user's  needs. 

« 

Even  thoupn  it  is  almost  trivial,  a final  comment  must  be  made  about 
the  importance  of  determining  the  objectives  of  various  potential  users 
followed  by  tne  analysis  of  the  users  present  system,  and  finally  the  syn- 
thesis of  all  parts  into  a design.  Considerable  effort  esn  be  avoided  by 
systematically  going  through  these  steps.  The  HFP  will  be  easier  to  write 
since  everyone  knews  exactly  wnat  is  needed.  After  having  accomplisned 
tnese  details  the  subsequent  evaluation  and  selection  process  should  be 
more  manageable. 
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J .2  Systems  Life  Evaluation  Costing  for  ADPZ 


3.2.1  Genera  1 

A good  comparative  cost  analysis  is  essential  to  the  AG?  selec- 
tion and  evaluation  process.  To  use  this  technique  properly,  it  is  neces- 
sary to  bring  tog-t..sr  all  costs  over  the  stated  systems  life.  In  addition, 
the  following  points  must  be  considered:  systems  life,  present  value  dis- 

count metnodology,  residual  value,  and  the  various  procurement  methods  a- 
vailable.  (12) 

» 

3*2.2  Systems  Life 

The  systems  (items)  life  must  be  established  by  the  Govern- 
ment based  upon  its  requirements  and  it  must  be  stipulated  in  the  Solic- 
itation Document.  The  "systems  or  items  life"  means  a forecast  or  pro- 
jection of  the  period  of  time  which  begins  with  tne  installation  of  the 
systems  or  item6  and  ends  wnen  tne  need  for  such  systems  or  items  na s ter- 
minated. systems  or  items  life  is  not  synonymous  with  actual  life  of  the 
equipment . 

3.2.3  Present  Value  Discount  .'■.etncdolory 

Tne  present  value  discount  methodology  as  set  forth  herein 
formalizes  a single  discount  rate  of  lC,c  for  all  DCD  agencies.  The  single 
rate  specified — 1C  per  cent — is  approximately  the  long-run  opportunity 
cost  of  capital  in  the  private  sector.  Under  this  concept,  the  payments 
made  over  time  will  be  adjusted  to  reflect  the  present  value  cf  those 
payments  a s of  the  data  of  contract  award.  Thus,  "all  expenses,"  while 
waiting  for  equipment  delivery  or  after  installation  for  the  stated  life  of 
tne  system,  must  bo  adjusted  to  reflect  present  value.  "All  expenses" 
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includes  net  only  tne  offeror's  prices  (equipment,  software,  and  support 
over  the  systems  life,  but  also  predetermined  in-nouse  expenses  for  A3?i 
installation  and  operation. 

Tne  following  formula  is  to  be  used  in  calculating  present 

value  cost: 

Expected  Discount  Present 

Montnly  X Factor  > Value 

for  IC% 


Cost 


Cost 


5 .2.4  Residual  Value 


Usually,  at  t ha  end  of  the  stated  systems  life,  the  computer 
system  still  h-*s  seme  value  to  the  Government.  This  value  nay  reflect  the 
fact  that  tne  initial  using  activity  may  well  keep  tne  system  longer  t.nen 
planned  or  some  otner  Government  activity  may  reutilize  the  hardware. 

Tne  future  lease  payments  saved,  js  well  as  the  resale  value  of  tne  equip- 
ment at  tne  end  of  tne  stated  systems  life,  affect  residual  value.  The 
residual  value  varies  witn-eacn  activity.  However,  for  general  purpose 
equipment  it  is  expected  t.nat  after  a five-year  systems  life  tne  equipment 
should  still  be  worth  approximately  2 C-5C,T  of  tne  purcnase  price  and  after 
eight  years,  about  1C, a.  The  following  formula  can  be  used  to  determine 
residual  values 

Purchase  price  * X X Present  Value  discount  = Residual 

factor  for  last  month  Value 
of  systems  life 

In  the  above  formula  "purchase  price"  is  tne  lowest  evaluated 
purcnase  price  offered  by  a responsible  and  responsive  offeror.  Any  pro- 
curement option  (e.g.,  Purcnase,  Lsase-to-Cwners.nip,  etc.)  t.nat  results  in 
the  Government  owning  t.ue  system(s)  will  have  the  residual  value  deducted 
from  tne  systems  life  cost  for  evaluation  purposes. 

* Including  the  operating  software,  if  priced  separately  from  the 
equipment  purcnase  price,  and  a perpetual  license  nas  been  ob- 
tained by  the  Government.  Instead  of  the  straight  purchase 
price,  tne  sum  of  all  invoice  payments  to  be  made  to  the  Con- 
tractor may  be  used  as  the  basis  for  this  calculation. 


5.2*5  Procurement  l-'.etnods 


Systems  life  costin?  should  be  calculated  for  each  procurement 
mstnod  offered.  Examples  of  t.ie  plans  currently  being  offered  are: 

a.  Purchase:  Outright  purchase  after  installation  and  ac- 

ceptance of  equipment. 

b.  Lease: 

(1)  Lease  aith  purchase  Option:  Lease  with  option  to 

purchase  at  predetermined  intervals  of  time.  The  purchase  price  is  usu- 
ally  reduced  by  subtracting  rental  credits  as  set  forth  in  the  offeror's 
proposal.  Purcnase  option  credits  greater  than  lCC^c  of  montnly  charges 
shall  not  be  considered  in  evaluating  offers  for  award. 

(2)  Long  Term  Lease:  such  plans  may  provide  multi-year 

leasing  at  determinable  prices  where  the  agency  exercises  a renewal  option 
at  tne  end  of  each  fiscal  year. 

(5)  Lea se-to-Cwnersnip  Plan  or  Lease  with  Title  Transfer 
Plan:  A plan  whereby  title  transfers  after  payment  of  n months  of  rental, 

but  usually  with  no  obligation,  or  less  obligation,  to  continue  to  lease 
than  in  (4)  below.  Normally,  title  transfer  does  not  occur  in  less  than 
six  years. 

(4)  Installment  purchase  Plan:  A plan  whereby  tne  Govern- 

ment exercises  an  option  to  purcnase  the  equipment  upon  payment  of  n 
montns  of  payment.  It  is  frequently  offered  as  a fixed  term  installment 
plan  usually  for  or  6c  months  in  whicn  tne  Government  either  is  granted 
title  immediately  or  title  is  passed  at  tne  end  of  tne  contract.  Normally, 
an  installment  purchase  plan  cannot  be  consummated  using  annual 
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appropriations.  Q»re  snould  be  tixen  during  nerotiations  to  ensure  tiiat 
tne  Government  retains  accrued  credits  and/or  equity  under  any  of  the  plane 
in  (b)  above,  snould  tne  agency  requirement  cease  or  funds  no  longer  be  a- 
vailable . 

5 .3  Alternative  Acquisition  Analysis 

It  is  very  important  tnat  an  analysis  of  the  alternative  methods  of 
acquisition  be  followed  witn  great  care  and  precision.  Frequently,  the 
same  vendor  will  net  be  low  on  betn  lease  and  purchase  plans.  Exhibits 
A - D on  the  following  pages  show  examples  of  different  lease  and  purchase 
plans,  as  computed  under  .the  present  value  discount  metacdolcgy . 
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1.  Tne  present  value  cost  is  determined  by  utinr  a discount  r-ts 
lCp  carried  to  six  decimal  places  and  assumes  end  of  the  year  costs.  7..J 
discount  factor  for  the  last  month  of  eacn  year  was  used  (i.e.,  12  mcnt.os 
•9C9C91*  24  montns  - .826446;  JC  mcntns  - .75151 ;;  4?  montns  - .6aJCl 
contns  - .62C?21;  72  months  - .564.474}  for  tne  sample  analysis.  In  an 

a c tua 1 economic  analysis,  the  preparer  would  use  the  applicable  discount 
factors  provided  by  tn'eir  deportment. 

2.  Six  year  systems  life. 

5«  Residual  Value  was  determined  as  follows; 


Purchase 

Price 


561C, CCO  X 20>  X 


Present  Value 
Discount  Factor 
for  last  montn 
of  systems  life 
.564474 


Residual 

Va  1 ue 
*65,866 


4.  The  Solicitation  Document  snould  clearly  state  how  the  purc.no s e 
option  will  be  evaluated  for  purposes  of  award. 
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Adjusted  Total  ~ $141,818  $426,144  $22,292  $20,2^5  $18,423  $1^,748  $61+5, 69O 

Less  Residual  Value  ($610,000  x 20$  x .564474)  68,866 
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1»  Tha  present  value  cost  is  determined  by  using  a discount  rate  of 
lCjl  carried  to  six  decimal  places  and  assumes  end  of  year  costs.  The  dis- 
count factor  for  tae  last  ncntn  of  eacn  year  was  used  (i.e.,  12  montns  - 
•9C9C91 ; 24  months  - .62c446;  months  - .751515;  4g  montns  - .6?5C15;  tC 

montns  - .62C921;  72  months  - .564^74)  for  tne  sample  analysis.  In  an 
a £ 1 economic  analysis,  tne  preparer  would  use  the  applicable  discount 
factors  provided  by  tn'eir  department. 

2.  Six  year  systems  life. 

5*  utraignt  six  year  lease;  residual  value  not  applicable. 
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3JK.VARI  OlG-fT  C?  aXdlBITS  A-D 
AS  GCi'^JiwD  wNLLh  I, in  PR-^fhliT 
VAL'Jn  DIjCT'JN:  ..ATHCSOLCOr 


Alternative  Methods  of 
Acquisition  Leuictsd 

A.  Lease  basis  with  option 
to  purcnase  (option 
exercised  at  the  end 

of  18  months) 

B.  Purchase  3a si 6 

‘‘I  \ 

C.  Lease  to  ownership  plan 
(Title  transfer  at  tne 
end  cf  six  years) 


Systems  Life  Cost 
nxhibits  A-D 
S576.82A 


$665,615 


;55S,2?i 


C.  itraignt  Six  fear  Lease  £574,896 

Tne  exhibits  are  designed  to  depict  the  alternative  aetncds  of  acquisition 
using  tne  present  value  discount  methodology.  No  attempt  has  been  made  in 
these  exhibits  to  identify  all  costs  associated  with  the  acquisition  of 
ADP£«  In  an  actual  economic  analysis,  cost  to  tne  Government  includes  not 
only  the  vendor  prices  (equipment,  software,  and  support). over  tne  systems 
life,  out  also  predictable  inhouse  expenses  for  ADPil  installation  and 
operation . 


45 


5 .A  Economic  Analysis 

An  economic  analysis  is  the  process  used  for  analysis  and  documenta- 
tion of  the  relationship  between  data  systems  and  the  functional  operations 
supported  by  tnese  systems.  (8,  C-l)  It  has  been  separated  into  discrete 
steps  for  convenience  of  presentation;  however,  it  will  be  noted  that  sig- 
nificant interplay  exists  among  these  steps  in  practical  application, 
while  it  is  realized  tnat  tnis  discussion  goes  beyond  the  computer  evalua- 
tion itself,  a manager  must  obviously  relate  it  to  the  overall  operation 
wnicn  it  is  to  support. 

Tne  following  information  concerns  items  of  interest  in  an  economic 
analysis . 

a.  Problem/opportunity  identification. 

The  problem  should  be  presented  in  terms  of  the  current  fu.net 
tional  deficiencies  which  are  targets  for  improvement  through  automation. 
Tne  initial  problem  statement  should  be  subjected  to  continuous  review  as 
part  of  the  ensuing  steps  in  tne  process,  since  it  will  undoubtedly  undergo 
several  major  changes  as  a result  of  tne  iterative  and  interactive  nature 
of  tne  process . 

b.  Relevant  environment. 

Describe  the  functional  environment  and  operations  that  tne 
proposed  system  is  to  support.  The  environment,  description  of  existing 
resources,  production  processes,  and  products  form  tne  baseline  alternative 
and  provides  tne  decisionmaker  and  analyst  witn  a common  point  of  depar- 
ture. Major  processes  and  associated  resources  in  the  area  of  functional 
operations  to  be  supported  by  tne  automated  data  system  should  be  identi- 


fied end  investirated 


c.  Objectives. 

Objectives  stated  should  be  specific  and  related  to  solvinr 
tnc  problems  or  realizing  t:.e  opport'ur.ities  identified  in  a above.  euphe- 
misms, sucn  as  "the  objective  of  the  proposal  is  to  provide  management  with 
core  accurate,  timely  information,"  are  not  applicable  objectives. 

d.  Assumptions  and  constraints. 

Assumptions  and  constraints  are  any  factors  that  limit  the 
flexibility  of  tne  decisionmaker  or  w.nicti  might  restrict  tne  use  of  the 
analysis  in  tne  decision  process.  Assumptions  focus  on  the  key  factors, 
processes,  and  variables  effecting  b..s  analysis,  but  which  are  not  explic- 
itly stated  in  ctner  parts  of  tne  analysis.  Constraints  are  factors  exter- 
nal to  tr.e  relevant  environment,  but  which  limit  tne  feasible  alternatives 
to  tne  problem  solution • 

e.  Al  terr.a  tives  . 

There  is  no  fixed  rule  on  the  number  of  alternatives  tnat  must 
be  considered.  However,  from  tne  discussion  below,  it  is  obvious  that  it  is 
highly  unlikely  that  tnere  will  ever  be  fewer  tr.an  two  alternatives  and  a 
baseline,  at  least  in  tne  earlier  phase  of  system  development.  Alternatives 
to  be  considered  involve  alternative  metnods  of  functional  operations  as 
well  as  alternative  methods  of  providing  automated  support  to  tnese  opera- 
tions . 

(1)  In  general,  the  selection  of  a computer  is  predicted  on 
using  AH?  to  correct  a shortcoming  or  improve  operations.  This  automatical- 
ly dictates  a baseline  (no  c mange  to  tne  present  fun:tior.'  and  two  other 
alternatives — one  wnicr.  requires  only  corre-ticn  of  the  snortcoming  (minor 
changes)  br.d  one  wnicn  envisions  major  cr.anres  in  eitner  the  „D?  or 


functional  procedure. 

(2)  Ctner  alternatives  which  must  be  addressed  as  appropriate 

a re » 

(a)  Use  of  existing  services  through  sharing,  consolida- 
tion, or  reutilization  of  Government-owned  facilities. 

(b)  Obtaining  similar  systems  or  ma^or  aspects  (appli- 
cations) of  similar  systems  from  other  DCD  or  Government  agencies. 

(c)  Use  of  contractor  support  (versus  in-house), 
f.  Costs. 

All  costs  should  be  identified  in  the  cost  portion  of  the  a - 
nalysis.  Costs  for  eacn  alternative  can  be  considered  to  fall  into  three 
broad  categories: 

(1)  Costs  within  tne  alternative  tnat  are  directly  related 
to  automatic  data  processing  (AD?)  support. 

(2)  Functional  costs  associated  with  the  alternative  that 
would  change  as  a result  of  implementing  tnat  alternative. 

(j)  Functional  costs  associated  with  eacn  alternative  as  an 
aid  to  visualizing  tne  costs  associated  with  each  alternative.  One  should 
consider  representing  the  relationship  of  two  alternatives  by  a diagram 
(fig.  }-l).  Let  a tria.np-le  represent  the  total  cost  of  alternative  A and  a 
rectangle  represent  the  total  cost  of  alternative  3.  AD?  related  costs 
should  be  identified  in  total  for  eacn  alternative.  Sasic  elements  of  r.D? 
related  costs  are  sno v..n  ir.  the  sample  forms  provided  at  figures  3-J,  3-^» 
and  J-p.  Tnese  cost  elements  are  used  to  describe  AD?  systems  in  tne  pre- 
gram/sudget  process  and  economic  analyses.  Tnese  costs  must  bo  portrayed 

Ut 


in  tne  analysis  in  sucn  a manner  tnat  if  tne  proposal  were  adopted,  tnese 
costs  would  be  identifiable  in  subsequent  prog ram/budret  documentation. 
Functional  costs  that  are  common  to  bctn  alternatives  (cross-hatched  area, 
figure  J-l)  need  not  be  incorporated  in  the  economic  analysis  since  they 
are  nondifferential.  The  last  category  of  costs  to  be  identified  is  func- 
tional costs  unique  to  a particular  alternative.  If  alternative  A is  the 
"no  change*  or  baseline  alternative,  the  area  labled  "functional  cost  u- 
nique  to  alternative  A,*  provides  the  basis  against  which  tne  unique  func- 
tional costs  of  eacn  of  the  otner  alternatives  will  be  compared.  Note  tnat 
the  alternative  may  oe  greater  than,  equal  to,  or  less  than  these  associated 
witn  tne  baseline  case.  In  all  circumstances,  eacn  alternative  will  be 
compared  to  tne  baseline  case  so  that  changes  are  identified  from  a common 
reference  point.  Identifying  cost  reductions  cr  decreases  in  relation  to 
the  baseline  case  constitutes  a commitment  by  tne  functional  manager  to 
absorb  corresponding  budget  reductions  at  specified  points  in  time.  In 
tnose  cases  wnere  functional  managers  plan  to  reutilize  resources  freed  by 
implementation  of  tne  KIS  ratner  than  absorb  reductions  in  that  area,  the 
specific  reutilization  must  be  identified.  For  those  functional  costs  that 
result  in  a net  increase  in  relation  to  the  baseline  case,  sufficient  de- 
tails must  ce  provided  to  facilitate  an  evaluation  of  the  impact  of  the 
increases.  The  analysis  for  eacn  alternative  considered  will  consist  of 
botn  Fart  I--A3?  nxpen&es  and  Part  II — Functiona 1 Expenses  in  the  formats 
as  shown  for  eucr.  phase  of  tne  life  cycle  of  eacn  alternative.  Figure  5-2 
grapnicaily  represents  the  distribution  of  cost  of  a baseline  and  one  pro- 
posed alternative  over  time.  Co3ts  rave  been  grouped  into  the  categories 

^7 


of  development,  phaseout,  and  operation  for  the  new  system.  These  cate- 
gories should  not  be  confused  witn  the  formal  phasing  of  the  system  life 
cycle  and  are  presented  only  to  illustrate  one  means  for  viewing  total 
systems  costs.  Some  costs  may  fall  into  all  three  categories  in  a given 
year.  Costs  which  have  been  expended  or  are  otnerwise  irrevocably  ooli- 
gated  to  a project  are  "sunk  costs"  and  should  be  identified,  but  excluded 
from  furtner  analysis,  i.e.,  not  included  in  tne  computation, 
g.  Benefits. 

(1)  Benefits  are  to  be  expressed  in  terms  of  dollsr  savings 
resulting  from  satisfying  tne  objectives  of  tne  functional  operations  sup- 
ported by  an  automated  data  system.  The  principal  task  to  be  'undertaken  in 
this  section  of  tne  analysis  is  to  isolate  tne  quantifiable  benefits,  in 
terms  of  tne  objectives  for  eacn  alternative.  In  many  cases  it  will  be 
first  necessary  to  estimate  or  measure  tne  change  in  systems  performance 
parameters  identified  as  demonstrating  satisfaction  of  the  system  objec- 
tives, tnen  subject  eacn  parameter  to  a detailed  analysis  of  the  value  of 
obtaining  that  amount  of  change. 

(2)  Data  to  support  the  benefit  analysis  must  be  identified 
early  in  the  life  cycle  of  tne  system  in  order  to  permit  adequate  prepara- 
tion, collection,  analysis,  and  coordination.  3aseline  data  snould  be 
collected  early,  witn  updates  as  required. 

n.  Compare  alternatives. 

(1)  The  comparison  of  alternatives  surfaces  tne  key  differ- 
ences, and  enaoles  tne  decisionmaker  to  focus  on  trade-offs.  The  basic 
procedures  for  these  comparisons  include  tne  use  of  present  value  tech- 
niques. Cnly  tnree  relaticnsnips  exist  as  concerns  the  cost  and  benefits 

U? 


of  various  alternatives:  Equal  benefits  Ain aqua  1 cost;  equal  cost/unequal 

benefits;  unequal  cost/unequal  benefits. 

(2)  The  unequal  ccst/unequal  benefits  case  is  both  tne  cost 
common  and  the  Boat  difficult.  Unless  every  possible  system  parameter  is 
identified,  measured,  evaluated,  and  translated  into  some  common  measure, 
tnere  is  no  all-purpose  criteria  for  identifying  tne  preferred  alternative. 

i.  Test  for  sensitivity. 

(1)  Derivation  of  costs  and  benefits  for  the  various  alter- 
natives under  consideration  Bay  have  been  acnieved  largely  throurh  tne  use 
of  sinulations,  projections,  and  assumptions  concerning  the  operational 
environment . No  matter  how  mucn  effort  has  been  invested  in  tne  assiduous 
use  of  tnese  techniques  to  obtain  en  accurate  portrayal  of  tne  environment, 
tne  result  stay  not  and  probably  will  net  be  the  same  as  that  environment. 
If  for  no  otner  reason,  the  future  orientation  of  the  analysis  interjects  a 
certain  amount  of  risk  or  uncertainty  into  the  results. 

(2)  The  major  job  in  this  step  of  tne  process  is  to  explore 

risks  and  uncertainties  witn  a view  to  discerninr  tne  potential  impact  of 
these  elements  on  tne  outcome  cf  tne  analysis.  This  is  accomplisned  by 
examining  the  key  cost,  benefit,  and  environmental  factors  and  relation- 
ships, in  light  of  variations  to  tne  stated  assumptions.  Two  analytical 
techniques  are  considered  useful  in  carrying  out  this  portion  of  the  anal- 
ysis: sensitivity  analysis  and  contingency  analysis. 

j.  Presentation  of  tne  analysis. 

The  completed  analysis  snould  be  structured  to  facilitate 
assimilation  and  understanding  on  the  part  of  reviewing  and  decisionmaking 


authorities . 


For  this  reason,  use  of  an  executive  sugary  and  appropriate 


graphic  displays  and  tables  are  useful  inclusions  to  the  basic  analysis 
document. 
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Figure  3-1 


Figure  3-2 
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Figure  3-5 


* «5  Request  for  Proposal 

Request  for  Proposal  must  cover  two  categories  of  items  for  any  AD? 
system.  Tne  first  category  concerns  those  items  wnich  are  mandatory.  This 
means  that  without  any  part  of  the  items  contained  -under  this  heading  the 
system  would  eit.ier  not  operate  or  the  entire  systam  would  drop  below  that 
which  is  considered  as  minimum.  The  second  category  covers  any  item  which 
is  not  included  in  the  mandatory  section.  These  are  considered  to  be  other 
system  requirements  (Cun's). 

To  determine  which  items  are  mandatory  and  which  are  CSR's  may  prove 
to  be  challenging  if  several  systems  have  been  developed.  In  this  case,  it 
may  be  best  to  compare  tne  configurations  and  designate  those  items  common 
to  all  systems  as  mandatory.  In  any  case,  the  mandatory  items  snould  in- 
clude only  tnose  items  without  which  the  planned  system  would  not  be  eit.ier 
operable  or  acceptable. 

The  CiR  items  should  be  listed  followed  by  a detailed  statement  con- 
cerning each  item.  A value  statement  must  be  prepared  which  shows  the  ra- 
tionale for  arriving  at  the  worth  of  tne  item  and  a template  which  snows 
how  tne  value  will  be  distributed.  More  details  concerning  this  method 
will  be  found  in  Lection  5-7»  selection. 

The  RF?  is  written  only  after  the  system  has  been  configured  and  tne 
mandatory/Curt  items  have  been  determined.  In  addition  the  cost  values  as- 
sociated with  the  C^R  items  must  iiave  been  est.blisued.  Several  approaches 
to  RF?  preparation  are  possible.  Da cn  method  witn  its  advantages  and  dis- 
advantages will  be  discussed  in  tne  following  tnree  sections. 


5.5*1  General  hr?  Specifications  fnauipmant  performance  Spec!  fi  ca  tic.os ' 
This  t yse  of  P.F?  is  most  probably  tha  easiest  for  the  user  to 
write.  This  metnod  involves  specifying  minimum  performance  requireirent3 
for  each  its:;,  such  as  memory  cycle  time,  disk  size  and  speed,  tape  speeds, 
anc  printer  speeds.  This  method  will' invariably  result  in  vendor  bias  if 
tae  specifications  are  not  cnosen  very  carefully.  Additionally,  this  meth- 
od say  not  allow  variances  in  individual  unit  speeds  that,  wnen  considered 
as  part  of  tne  wnole  process,  meet  the  overall  system  requirement.  How- 
ever, if  the  solicitation  specifies  performance  with  bencnmark,  a fair  and 
equitable  procurement  snculd  result,  .v i t h this  approach,  tne  vendor  de- 
signs tne  largest  portion  of  the  system,  thus  allowing  him  tne  freedom  to 
configure  as  he  pleases,  using  tne  few  constraints  associated  with  the 
given  files.  The  merit  of  this  method  lies  in  the  fact  that  tne  vendor's 
analyst,  hopefully  with  broad  experience,  will  design  the  best  possible 
system,  as  Joslin  indicates,  it  exposes  the  organization  computer  require- 
ments to  tne  top  vendor  analysts,  (lpc,  9) 

There  are  several  problems  connected  with  writing  a general 
hFP . Due  to  its  nature,  tne  user  can  expect  to  spend  many  hours  witn  the 
vendor  clarifying  tne  intent  of  tne  various  files  mode  of  operation  plus  a 
host  of  otner  variables.  Ine  user  is  also  likely  to  spend  many  more  hours 
trying  to  verify  the  systems  proposed  by  tne  vendors  to  ensure  that  they 
will  work.  The  prospective  user  must  eitner  do  tnis  or  run  tne  risk  of 
buying  a system  that  will  not  quite  be  capable  of  handling  the  applications 
it  is  required  to  handle.  Also,  it  is  very  important  that  the  prospective 
user  thorournly  understand  tne  system  concepts  whicn  tne  vendor  proposes, 
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because  no  cotter  wni.cn  system  he  selects,  the  vendor's  representative  who 
wrote  the  proposal  and  suggested  the  concept  will  not  be  delivered  with  the 
system.  It  will  be  ine  responsibility  of  the  user  to  turn  the  concept  into 
reality.  Depending  on  the  amount  of  work  accocplisned  in  tne  analysis 
phase,  the  evaluation  process  can  be  either  short  and  smooth  or  long  and 
difficult.  In  essence,  it  is  often  difficult  to  compare  competitive  pro- 
posals and  to  select  cna  above  all  the  others. 

5.p.2  Retailed  specifications  (Data  bystems  Specifications) 

Ine  greatest  overall  advantage  in  writing  sucn  an  RF?  is  the 
fact  tne t detailed  tnougat  processes  whicn  must  ro  into  eacn  step.  Tne  user 

» 

is  forced  to  inspect  his  entire  system,  he  must  spell  out  eacn  step  to  oe 
taken  in  each  of  tne  applications.  Tnis  metnod  includes  establishing  the 
objective  of  tne  system  and  presenting  t.ne  data  processing  requi rements  for 
accomplishing  the  objectives.  The  followinr  data  are  required:  (8,  K-5) 

a.  Througuput  requirements 

b.  File  description,  record  sire,  etc. 

c.  Transaction  volume  and  descriptions 

d.  Card  and  printer  l/C  volumes 

e.  Terminal  I/O  volumes 

f.  Information  on  sequence  requirements 

g.  Timing  or  turnaround  restrictions,  etc. 

Tne  prospective  user  must  ta/ce  great  care  in  writinr  detailed  specifics  tions 
to  ensure  tnat  tney  do  not  become  ma cnine-orier.ted  ratner  than  application- 
oriented.  i.acnine-oriented  speci  fi  ca  tion3  may  discriminate  against  some 

, and  tnus  unintentionally  keep  tne  company  from  getting  the  system 
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that  would  beat  meet  its  needs.  Detailed  specifications  require  tne  ven- 
dors to  configure  their  systems  exactly  as  demanded  by  the  specifications . 
This  simplifies  systems  design  work  for  tne  venders,  but  allows  tr.em  little, 
if  any,  freedom  to  fit  tne  applications  to  their  computers.  Ratner  the  com- 
puters must  be  fitted  to  tne  applications.  Tne  advantage  here  is  quite  ob- 
vious. Ine  user  detects  omissions,  errors  in  description,  ar.d  inconsist- 
encies . 

Detailed  specifications  offer  anotner  definite  advantage  to  tne  user 
in  that  tney  descrioe  tne  situation  completely  and  tney  define  each  appli- 
cation fully  and  uniformly  to  all  venders.  Thus  tne  user  has  to  waste 
little  time  in  talking  to  vendors.  when  the  proposals  are  submitted,  the 
user  can  more  easily  verify,  compare,  and  evaluate  them,  since  the  systems 
proposed  must  all  be  identical  to  steps  set  fortn  in  the  specifications. 

No  system  will  be  proposed  that  is  far  inferior  to  the  system  demanded  by 
the  specifications.  On  tne  otner  hand,  no  system  will  be  proposed  that  is 
far  superior  to  tne  system  specified,  eitner. 

Tnis  method,  newever,  also  nas  its  drawbacks.  The  time  and  effort 
expended  in  the  writing  may  be  too  costly  for  tne  system  whicn  is  to  be 
acquired,  anotner  major  point  is  tne  fact  that  tne  detailed  request  allows 
tne  vendor  little  improvement  on  the  basic  design.  As  Kr . Joslin  stated, 
"linen  detailed  specifications  are  used,  the  system  proposed  will  be  no 
better  tnan  tne  system  riescrioed  in  the  specifications,  but  the  trouble 
involved  in  obtaining  tne  system  is  minimized."  (ljc,  11 ) 
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the  prespective  user  will  be  doing 
will  discover  any  proolem  areas  tn 
cations  to  tne  vendors.  Inis  natu 
tion6  between  vendors  and  user. 

Proposals  submitted  in 
tion  should  all  present  solutions 
forth  in  tne  detailed  specificatio 
inative  cr  exceptional  proposal, 
should  be  scmewr.at  simpler  than  vs 
suiting  from  straigat  general  spec 
main  objectives  of  using  ccmbinati 
problem  of  proper  verification  and 
little  importance  compared  with  oo 
puter  systems. 

Frequently  tne  type  of 
system  indicates  tne  best  type  of 
pose  tnat  tne  application  is  a sci 
problems  cf  a relatively  fixed  na t 
cost  satisfactory,  since  there  is 
the  computer  in  tne  system.  Cn  th 
calls  for  many  users  to  time-snare 
menta  would  constantly  vary,  so  tn 
tailed  specifications  for  sucn  a s 
desirable  even  if  tney  could  be  pr 
prospective  user  is  really  seedin’ 


The  vendor's  responses  snould  follow  a preselected  ferns t 


Tne 


Government  services  Administration1 s Guidance  on  tne  Preparation  of  specifi- 
cations, ue lection  and  Acquisition  cf  Automatic  Data  processing-  systems  of 
27  August  197c  offers  excellent  advice  on  hr?  content.  Th_  exnaustive  lists 
offered  for  ,-iF?  preparation,  oenchma rkine,  etc.,  are  very  good.  Tne  pro- 
spective user  snould  ensure  that  he  reviews  this  document  in  considerable 
detail.  In  addition,  tne  same  document  provides  a "solicitation  Document 
for  aD?  systems"  wnicn,  in  essence,  is  a model  a DP  contract.  The  provisions 
tnerein  apply  to  all  federal  agencies. 

5.5.^  request  For  rropcsal  Contents 

The  r.F?  is  tne  key  to  a good  user-vendor  relationship.  If  tne 
user  is  willing  to  accept  a response  of  minimum  quality  from  vendors,  tne 
nF?  mignt  contain  merely  a description  of  tne  system  specifications,  a few 
statements  about  necessary  vendor  support  of  tne  system,  and  the  due  dates 
for  tne  submission  of  proposals,  nuen  a bid  request  might  suffice  for  tne 
purpose,  but  it  certainly  does  not  constitute  effective  contact  witn  the 
vendor,  a good  r.F?  snould  contain,  at  the  very  least,  statements  concern- 
ing tne  following  elements:  (l*c,  11?) 

a.  system  requirements;  This  section  saculd  contain  the  re- 
quirements of  tne  expected  system. 

b.  Vendor's  support  of  system:  This  section  snould  contain 

statements  about  expected  training,  back-up  facilities,  manpower  taat  tne 
vendor  is  expected  to  supply,  etc. 

c.  Tecunical  questionnaire  and  timing  taoles:  These  question- 

naires and  taoles  seek  to  elicit  inforxition  on  r.ordware,  software,  vendor 


62 


support,  and  equipment  ccst,  as  hall  as  data  on  timing.  Trie  prospective 
user  snculd  request  tnat  tne  vendor  rive  references  to  the  vendor  literature 
or  to  otr.er  tecr.r.icul  sources  in  support  cf  all  information  offered  in  an- 
swer to  tne  questions  as.red  in  the  questionnaire.  Tne  user  should  also  re- 
quest detailed  information  to  suostantiate  all  the  vendor's  timing  estimates. 
Tne  questionnaire  and  tne  timing  tables  should  be  preceded  by  a concise  and 
forthrignt  explanation  of  now  tne  information  that  tne  vendors  furnisn  will 
be  used.  Tne  vendor  will  taus  gain  a greater  understanding  of  tne  ques- 
tions asked,  and  he  will  be  better  able  to  supply  the  information  requested. 
But  more  important  is  the  fact  that  t.ie  vendor  will  acquire  a degree  of  in- 
eignt  tnat  will  permit  nix  to  recognize  tne  intended  meanings  of  questions 
tnat  might  otherwise  nave  seemed  ooscure  to  nim.  By  explaining  the  use  to 
which  ns  intends  to  put  tne  answers,  and  way  ne  is  asking  tne  questions,  the 
user  is  in  essence  telling  tne  vendor  how  ne  is  going  to  evaluate  the  pro- 
posal, wnat  factors  are  important,  and  what  relative  importance  eacn  factor 
ns  s • 

d.  3enc:uiark  data:  If  benchmarks  are  to  be  used,  the  data  for 

these  benchmarks  should  be  supplied. 

e.  Bidder's  conference  data:  If  the  system  is  complex  or  very 

costly,  a conference  sr.ould  oe  scheduled  to  discuss  the  applications  cov- 
ered in  tne  proposal  as  well  as  tne  evaluation  methods  to  be  used.  Tne 
chosen  data  snouid  give  tne  potential  venders  encugn  time  to  prepare  for 
tne  meeting. 

f.  Dates  of  significant  events;  a11  critical  dates  such  as 
proposal  due  date,  awarding  date,  system  installation  date,  etc.,  should  be 


included  in  this  section 


g.  Provisions  for  handling  questions}  wince  questions  must  be 
expected  after  t.ne  proposal  nas  been  released,  a central  candidate  is  the 
Procurement  Officer.  No  memoar(s}  cf  t.ie  evaluation  team  should  be  in- 
volved since  prejudices,  both  pro  and  con,  toward  a particular  vendor  nay. 
ensue . 

h.  Vendor  denonstra tion:  Benchmarks,  if  any,  are  usually  ex- 

ecuted during  this  deccnstra tion  period. 

The  above  items  concern  tne  more  tec.nnical  AD?  aspects  of  tne 
RFr.  To  enable  an  of feror/oiddor  to  prepare  a proposal  or  quotation,  tne 
solicitation  will  identify  all  tne  evaluation  factors  tnat  are  tc  be  con- 
sidered. Tne  Armed  cervices  Procurement  Regulation  prescribes  tne  specific 
format  and  annexes  required  for  any  RFP.  These  aspects  obviously  must  be 
followed  in  preparing  such  a document.  The  sole  intent  of  tne  above  *03 
to  dwell  on  AD?  peculiar  items. 
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3 .6  Validation  of  the  Proposed  bystens 

<»hy  validate?  kr.  Joslin  writes  tnat,  "tne  user  should  assume  that 

fH 

the  claims  c onta ined^tns  vendors'  proposals  cannot  be  accepted  at  face 
value  . . . the  prospective  user  will  find  tnat  a vendor  interpret*  state- 
ments and  requirements  so  as  to  favor  his  own  equipment  ....  Tne  ven- 
dor . . . may  greatly  expand  and  enlarge  favorable  answers  to  any  question 
wnose  answers  cannot  be  factually  or  physically  proved  or  disapproved.* 
(13c,  63)  Although  less  blunt,  Mr.  Timmreck  points  to  essentially  the  same 
problems,  he  states  that:  "The  vendor's  objective  . . . is  to  maximize 

his  profits  ....  He  will  often  tend  to  empnasize  certain  features  with 
respect  to  which  his  machine  is  superior  to  others.  . . . Certain  aspects 
of  his  Hardware  and  software  will  be  designed  to  'lock  in'  tne  user,  tnat 
is,  to  make  conversion  away  from  tne  vender's  equipment  prohibitively  expen 
sive.  It  seems,  then,  tnat  the  ancient  adage,  'Let  tne  buyer  beware'  holds 
in  tne  purcr.ase  of  multimillion  dollar  equipment  as  well  as  in  tne  grocery 
ma  rke  t . * 

These  basic  comments  lead  to  tne  next  question.  How  is  a given  system 
validated?  nre  there  any  guarantees  tnat  tne  selected  system  will  maximize 
tne  stated  objectives  wnile  minimizing  the  expenses  incurred?  beverel  metn 
ods  will  be  described  and  critiqued  in  the  following  pages.  Some  of  tnese 
metnods  are  currently  in  use;  some  have  been  discarded,  while  others  are 
still  being  furtner  developed  and  refined. 

5*6. 1 Mandatory  Hepuirements 

Validation  of  the  mandatory  requirements  is  a rather  quick  and 
painless  procedure--the  items  and  their  specifications,  as  listed  in  the 
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RFP,  are  checked  against  each  vendor's  proposal.  Additionally  an  in-house 
review  snould  oe  made  using  current  tecnnical  data  supplied  by  vendors,  AD? 
Industry  standard  reports,  etc.,  to  validate  all  aspects  of  the  proposal. 
Also  oral  presentation  snould  be  sought  from  bidding  vendors  to  facilitate 
elaboration  of  tneir  proposals.  In  this  case  members  of  the  Source  Delec- 
tion Evaluation  3oard,  9fter  reviewing  the  proposals,  will  be  expected  to 
attend  each  vendor's  presentation.  Lastly,  those  items  wnich  cannot  be 
directly  validated  sucn  as  a vendor's  promise  to  supply  a given  support 
item,  etc.,  should  be  very  clearly  defined  in  the  contract.  If  trie  ven- 
dor's bid  does  not  meet  any  one  of  the  mandatory  requirements,  suspect  to 
the  above  discussion,  tnat  proposal  is  immediately  disqualified. 

3.C.2  System  Timin* 

Unfortunately,  the  matter  of  system  timing  is  not  simple  or 
easy,  ‘-.any  articles  are  available  to  tne  reader  of  computer  literature 
wnich  describe  various  timing  methods  and  processes.  ,.ith  tne  help  of 
tnese  metnodologies,  tne  user  snould  be  able  to  determine  whether  or  not 
the  system  will  process  tne  data  in  a specific  amount  of  time. 

3asically,  every  system  timing  method  relates  directly  to  the 
workload  whicn  tne  ayatem  is  expected  to  process,  rhetors  such  as  equipment 
and  storage  usare,  lar.ruage,  order  or  sequence  of  'obs  executed,  and  tasks 
of  tne  type  which  are  expected  to  be  processed  must  be  taken  into  consider- 
ation. Many  methods  fer  constructing  a drive  workload  have  appeared  with- 
in  tne  recent  past. 

Joslin  ( l^a , 27-57)  describes  a technique  for  selecting  repre- 
sentative benchmarks  by  classifying  tne  workload  according  to  the  type  of 
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Job  and  tnen  selectir.r  a combination  of  Jobs  to  represent  the  character- 
istics of  each  class,  iir.ope  et  al.,  constructed  a drive  workload  consist- 
ing of  Jobs  selected  from  tne  actual  workload  by  statistical  sampling  and 
tnen  adjusted  this  collection  until  it  was  considered  representative.  sood 
and  Foreman  (2P,  yl-55)  composed  a syntnatic  workload  using  Shope's  technique 
and  substituting  3ucnnolz's  (4f  JC°-5lP)  syntnstic  program  for  eacn  Job  in 
tne  mix.  Tne  parameters  of  the  synthetic  program  seem  to  have  been  deter- 
mined by  trial  and  error.  Recent  proposals  for  creating  an  industry-wide 
library  of  standardized  bonc.nmarns  (a.  C.  .ucas)  (ly,  1C41-1C5?)  and  a li- 
brary of  synthetic  modules  (h.  1^.  Gumse)  (11,  IC)  may  reduce  the  number  of 
mannours  required  to  cede  and  debug  the  programs,  but  tnere  still  remains 
tne  need  for  a metnod  of  constructing  a representative  workload  from  such 
a collection.  Ferrari  (1C,  13-24)  stresses  the  importance  of  workload 
cnara cterization  and  points  out  tne  need  for  methods  cf  constructing  drive 
workloads  tnat  are  representative  of  real  workloads.  Previous  methods  cf 
Job  selection  may  be  convenient,  but  they  may  be  somewhat  arbitrary  and 
inaccurate.  (24,  2) 

Although  eacn  of  the  above  methods  differs  somewhat  in  construc- 
tion, tneir  common  element,  eitr.er  stated  or  implied,  is  the  concept  of 
"representativeness"  of  tne  total  cnaracteristics  of  any  workload  which  is 
expected  to  drive  tne  system,  aven  though  tne  analysis  of  the  present  sys- 
tem has  been  painstakingly  executed,  tnere  is  little  dcubt  tnat  tne  compi- 
lation cf  a representative  drive  workload  with  all  the  cnaracteristics  cf 
tr.e  "real  world"  is  tne  .ost  difficult  phase  in  any  evaluation  ar.d  selec- 
tion of  a computer  system.  The  issue  cf  representative  drive  workloads 
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will  be  addressed  again  wnen  " bencnma  rking*  is  discussed.  Cnee  the  drive 
workload  has  been  built  (by  any  ol'  tne  above  methods)  tne  system  can  be 

timed . 


a . Add-Time  Comparisons 

Add-time  comparisons  were  used  in  the  past  to  measure  cos 
puting  power  between  two  macnines.'  This  method  was  abandoned  due  to; 

(1)  Variables  in  adder  circuitry  and  core  storage  access 


tire  . 


(2)  -ingle  versus  multiple-address  organization  tire  was 
not  taken  into  account. 


(5)  Tne  metnod  completely  ignored  any  consideration  of 
software.  (24,  12) 

b . instruction  i.ixes 

This  metnod  attempts  to  broaden  tne  range  of  tne  evaluation. 
The  mix  of  instructions  is  determined  by  the  us;r's  application  programs, 
mach  of  the  mixes  is  simply  a weighted  average  of  the  execution  times  for  a 
number  of  tne  most  commonly  used  instructions.  A weighting  factor  is  as- 
signed to  eacn  instruction  in  accordance  with  someone’s  opinion  of  that  in- 
struction's frequency  of  occurrence  in  programs  of  a certain  general  type. 
The  evaluation  is  based  on  multiplying  the  weight  factor  by  tr.e  manufac- 
turer's specifications  for  that  instruction  and  summing  each  total.  Al- 
though more  instructions  are  used  by  this  metnod,  cany  of  tne  shortcomings 
of  tne  add-time  remained,  several  ct:.er  difficulties  exist  in  this  metnod. 

(1)  I/C  considerations  and  compiler  efficiency  is  usually 


ignored,  (p,  ^1) 


Instruction  overlap  facilities  are  hard  to  reflect 


(2) 


accura  tely . 

(3)  Tne  result  obt9ir.sd--a  v.ei  rated -average  instruction 
time--mey  be  a fair  representation  for  an  individual  system  in  a particu- 
lar application  area;  however,  it  is  Bluest  meaningless  to  use  in  a com- 
parison when  tne  number  of  insturcticns  required  by  each  system  is  not 
known.  (1,  13) 


(A)  No  generally  accepted  criterion  for  determining  the 
weights  has  been  established. 

(5)  Although  the  instruction  sets  differ  among  machines, 
eech  must  be  weighted  the  same. 

(6)  In  general,  the  instruction  mix  method  is  inapprepriat 
for  selection  purposes  end  inapplicable  for  software  evaluation. 

c.  Kernels 

A kernel  is  a small  routine,  which  is  usually  quite  simple. 
Further,  it  is  ceded  in  the  particular  machine's  own  language.  This  method 
was  quite  an  advance  over  the  instruction  mix  procedure,  home  advantares 

were : 


(1)  The  timings  are  Dased  on  the  manufacturer's  stated 
execution  time  for  tr.e  instructions  that  are  included  in  the  kernel. 

(2)  iir.ee  the  process  uses  the  machine's  own  instruction 
set,  characteristics  unique  to  a machine  could  be  fully  exploited  to  the 
vendor's  advantage. 

(3)  The  kernel  can  include  more  parameters  than  the  pre- 
vious techniques.  (15,  52) 
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Lome  innerer.t  di  .advantages,  however,  began  to  appear  when  multiprogram- 
ming and  multiprocessing  were  used.  Tnese  are: 

(1)  Kernels  are  renerally  isolated  tasks,  tnat  is,  a ker- 
nel cry  represent  the  cain  loop  of  a.n  application  prcgram. 

(m}  In  conjunction  with  tne  aforecentioned , I/O  is  not 
properly  ceasured  s.nce  tne  stress  is  on  an  instruction  set. 

(5)  Modern  systems  exnibit  great  overlap  and  simultaneity 
of  events  and  thus  a dependence  upon  the  sequence  and  requirements  of  tne 
total  processing  workload.  (22,  6) 

(A)  There  is  net  a sinrle  typical  kernel,  and  further 
there  are  no  accepted  or  standardized  weights  for  combininr  kernels.  Much  t 
effort  is  required  to  cede  and  time  larre  numbers  of  kernels.  It  is  also 
difficult  to  determine  if  equal  programming  skills  were  used  for  each  ker- 

J 

nel.  '1=  2 

In  general,  kernels  have  been  abandoned  for  comparative 
evaluations.  The  relative  power  of  a system  is  net  necessarily  how  fast 
it  is  internally,  but  now  fast  it  can  perform  tne  complete  job.  (2f,  2C?l} 

In  any  thrournput  evaluations  or.e  must  consider  tne  interaction  of  internal 

performance,  with  I/O  speeds  and  facilities,  in  addition  to  tne  cost  iepor-  1 

tant  factor  of  programming  systems  efficiency.  (1,  14} 

’Jp  to  this  point,  methods  of  evaluating  pure  processing 
power  have  been  discussed.  Due  to  their  inherent  snortcominrs,  they  neve 
virtually  disappeared  from  computer  evaluation  metnods.  "Benchmarking"  has 
taken  piece.  The  next  section  will  discuss  the  different  types  of  bench- 
marks and  the  pros  and  ccr.s  of  eucn  type. 
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d.  3er.chma  rks 


(1)  General 

3enchmark  job  streams  have  been  used  with  increasing 
frequency  for  tne  purpose  of  improving  the  present  performance  of  a system, 
predicting  tne  effects  of  changing  workloads  on  a system,  or  sizing  and 
selecting  a new  computer  system.  There  must  be  a drive  workload  that  imi- 
tates tne  actual  workload  with  reasonable  fidelity,  workload  is  defined  as 
the  collection  of  all  individual  jobs  and  data  tnat  are  processed  by  the 
computer  system  during  a specified  period  of  time.  (24,  1)  Cne  mignt  de- 
fine bencnmarks  as  follows:  Tney  are  mix  (or  grouping)  of  routine  to  be 

run  on  several  different  computer  configurations  in  order  to  obtain  compar- 
ative thruput  performance  figures  on  the  capabilities  of  the  various  con- 
figurations to  handle  the  specific  applications.  This  definition  of  bencn- 
marks includes  the  following  three  key  cha ra cteri sti cs : first,  tne  routines 

are  to  be  actually  run  on  the  configuration ; second,  the  total  throughput 
time  is  important  (not  just  processor  time);  and  tnird,  tney  are  aimed  at 
specific  applications. 

Tne  concept  of  "representativeness*  of  tne  drive  work- 
load has  been  discussed.  To  meet  tne  above-cited  criteria,  tnree  types  of 
bencnmarks  nave  been  used: 

(a)  Use  of  tne  real  workload  (live  benchmark)  without 


cnange . 


(b)  Design  a workload  (artificial  bencnmark)  indepen- 


dent of  the  real  workload. 

(c)  Assemble  a workload  (hybrid  benchmark)  from  parts 


of  tne  real  workload. 


71 


(2'  Live  Benchmarks 

Tne  simplest  and  least  costly  met..od  is  to  ran  tne 
present  workload  against  tne  proposed  system.  No  problems  will  oe  encoun- 
tered with  "representativeness"  nor  witn  the  relationships  between  real 
workloads  and  benc.ima rks . Inis  method  i3  juite  acceptable  if  tne  new  sys- 
tem is  upgrading  an  existing  one.  mven  this  "ideal*  approach  has  its  dis- 
advantages . 

(a)  Since  the  live  workload  is  tne  system  and  the  de- 
mands on  the  system  change  within  a given  time  frame,  how  do  you  chocse  and 
group  the  typical  workload  so  that  they  will  give  a fair  picture  of  tne 
system's  workload?  Tc  overcome  this  problem,  Mr.  Joslin  gives  a detailed, 
cookbook  approach  designed  to  assure  that  tne  bencnmark  is  truly  represen- 
tative. ( 1 J c , t£-£l) 

(b)  Jnless  tne  production  programs  are  written  in  a 
higner  level  language,  t.te  entire  sc:. erne  of  a natural  or  live  oencnmark  will 
net  work  since  tne  programs  will  not  be  portable,  (l*f,  27?) 

(c)  Benchmarks  are  prepared  and  processed  usir.e  a 
variety  of  procedures  resulting  in  unduly  long  execution  times,  unreason- 
able file  volumes,  end  inconsistent  measurement  procedure.  (IS,  A) 

(d)  Live  benchmarks  are  usually  associated  with  high 
costs,  both  to  tne  buyer  Bnd  vender,  in  terms  of  time  and  money.  (1?,  5) 
Kuch  of  this  time  is  spent  by  tne  vendor  in  adjusting  tne  bencnmark  so  that 
it  will  satisfactorily  run  on  his  proposed  system. 

(})  Artificial  Benchmarks 

An  artificial  bencnmark  is  a prorram  which  models  a 
live  bene. .mark,  several  metnods  of  artificial  jenc. -marks  nave  already  been 


72 


discussed  in  detail.  Instruction  nixes  were  considered  representative  of 
the  major  functions  of  that  system.  The  kernel  method  improved  on  tne  in- 
struction nixes,  as  an  example,  by  taking  I/C  activity  and  CPU  utilization 
into  consideration.  Tr.s  drawbacks  to  tnese  metnods  were  discussed  in  the 
previous  section. 

(4)  Hybrid  Benchmarks 

As  the  term  implies,  a hybrid  benchmark  is  a combina- 
tion of  metnods  used  to  evaluate  a proposed  system.  In  most  cases,  live 
benchmarks  ferm  the  core  of  the  process.  Synthetic  programs  may  then  be 
used  tc  exercise  tne  machine  if  it  is  felt  that  the  live  data  will  prove 
to  be  inadequate.  Bet.,  may  be  supplemented  by  simulation,  particularly  in 
tnose  instances  wr.ere  remote  sites  or  telecommunications  are  involved.  a1 
thou-a  this  approacn  seems  tc  cover  as  many  areas  as  possible,  tr.e  disad- 
vantages are  again  found  in  tne  rela tionsuip  o:  the  benenmark  to  tne  real 
and  total  workload.  Tne  problem  encountered  in  synthetic  programs  and 
simulation  will  be  discussed  next. 

e.  Synt.oetic  Programs  as  Ben cnmar.cs 

A synthetic  program  may  bs  defined  as  a job  wncse  usage  of 
various  resources  may  be  parametrically  changed  to  fit  t.ns  characteristics 
of  a rs->l  workload.  The  programs  may  be  sorted  in  two  groups: 

(1)  The  syntactic  pre.-ram,  w.nich  is  to  represent  a real 
program,  is  designated  as  the  task-oriented  element. 

(2)  These  syr.tnetic  programs  which  can  be  adjusted  to  uti 
lize  various  amounts  of  CPJ  time,  I/C  time,  etc.,  are  defined  as  resource- 


oriented  elements 
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The  building  of  a synthetic  bencnmark  is  based  on  a "repre- 
sentative" sairple  of  tno  workload  and  production  progress  of  the  present 
systes.  Their  characteristics — I/C  channel  activity,  CPU  tise , sesory 
space — are  seasured.  A synthetic  job  or  jobs  is  then  built,  using  tne  de- 
tersi.ned  cnara  cteristics  of  tr.e  real  enviror.sent.  The  goal  is  to  rescve 
cosplexity  while  preserving  sensitivity.  (22,  ?) 

The  principle  reason  for  using  syntnetic  programs  is  their 
flexibility.  They  can  be  made  machine  independent  and  portable  because 
t.oey  ate  written  in  a nigher  level  language.  Still,  problems  do  exist. 

Cne  reus  or.  fcr  this  might  oe  t:;e  fuel  t .a  * tr.e  characterization  of  the  real 
workload  used  to  set  the  parameters  of  the  syntnetic  job  is  not  adequate. 

In  essence,  It  does  not  uniquely  determine  the  rarfcrmence  variables  tnat 
are  to  be  measured.  (1C,  22'  David  . Lambert  cites  t..e  followin'  problem: 

(1}  the  lack  of  standard  terms  and  measures  across  CJcnir.es;  (e.)  workload 
determination  and  repres-ntutic.o  for  conceptual  systems;  (})  transferability 
of  syntnetic  programs;  (A)  workload  specification  and  generation  for  on-line 

I 

systems;  and  (5)  effects  of  new  computer  system  architectures  on  present 
cetnods.  Pernaps  Oliver  has  stated  tne  problem  with  syntnetic  programs  most 
succinctly:  "It  is  interesting  to  note  tnat  all  suggestions  on  how  to  ncdel 

a workload  rely  on  one  of  tne  evaluation  teehni-ues  surveyed  . . . ^monitors, 
simulation,  etc.'.  Thus  we  s.nould  not  expect  the  synthetic  mix  approach  to 
be  an  improvement  over  t.nese."  (IP,  1 5) 
f . Sicula  tion 

A simulator  is  a pro -rum  paexare  whica  represents  tne actual 
machine  under  consideration.  This  proves  very  useful  for  evalu.ting  a co.n- 
c " .si  ay-t-m.  ».r . uo.nry  C.  Lucas  culls  si-ulatior.  "tne  most  cote  -tia  ily 
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application  since  it  is  aimed  at  trie  13..'.  }6c  system.  within  this  con- 
straint, however,  C..  is  a powerful  tool  for  the  modeling  cf  computer  sys- 
tems. The  language  addresses  itself  to  the  equipment  and  its  corresponding 
characteristics.  It  is  generally  used  to  determine  t.ie  impact  cf  new  equip 
ment  on  tne  system,  cnanges  in  the  configuration,  etc.  Cu_  training  is 
offered  by  IBM  to  its  customers  at  no  extra  cost. 

(b)  Many  simulators  which  use  these  and  other  lan- 
guages are  now  commercially  available.  They  will  aid  tna  user  in  evaluatin 
his  current  system  or  in  evaluating  tne  configurations  which  are  bid  by  yen 
dors.  A description  of  two  popular  simulators  is  provided  below. 

(l_)  Systems  and  Computer  -valuation  and  Review 


Tecnnique  (SCmRT) 


SCuRT  is  e proprietary  simulation  package 
developed  by  Ccmress,  Inc.,  of  Wasningtcn,  C.  I c.i  simulator  has  seen 
used  in  a large  variety  oX'  applications  i.nvolvinr  xessioili ty  analysis, 
equipment  selection  studies,  system  und  prcrram  desi-n  and  system  mainte- 
nance. aCmni's  main  advantage  lies  in  t.ie  fact  tr.at  it  does  not  utilize  a 
dedicated  language  sucn  as  &?t_  or  Cu~  since  the  application  system  defini- 
tion 8nd  tne  hardware/software  definitions  are  inde oendent . This  is  quite 
significant  in  proposal  evaluation.  By  holding  the  system  definition  con- 
stant, tne  equipment  configuration  can  be  chanred  for  each  vendor.  During 
St 

the  system  design  stac-e,  the  converse  holds.  The  C.ZRT  package  contains 
algorithms  for  optimizing  numerous  aspects  of  the  total  system.  This  in- 
cludes tne  blocking  of  records  on  tape  or  mass  stcrare  scheduling  in  a 
multi-programmed  environment  ->r.d  tne  astirnment  of  peripherals  to  I/C 


(2) 


Computer-Aided  System  Evaluation  (JAom) 

CnS£  is  a proprietary  package  developed  and 
marketed  by  software  products  of  Falls  Church,  Virginia.  The  package  v. i 1 1 
simulate  all  types  of  systems  for  any  manufacturer.  The  c.nief  adva  r.ta  -e 
of  CrthE  is  tne  program's  automatic  system  design  feature.  This  is  quite 
helpful  during  the  design  and  evaluation  of  a new  system,  dy  specifying 
the  workload  and  the  desired  configuration,  a system  design  will  be  pro- 
duced which  is  independent  of  any  one  type  of  machine. 

(c)  Simulators  have  been  used  to  redesign  file  struc 
tures,  test  tr.e  utilization  of  alternate  I/O  devices,  increase  memory  ca- 
pacity by  pointing  to  redundancies  and  test  varicus  workloads  on  various 
configurations.  Further,  many  packages  are  commercially  available  wnich 
are  maintained  with  extensive  libraries  of  manufacturer  data  for  many  con- 
figurations. Finally,  simulators  have  been  extensively  used  in  formal 
competitive  procurements . 

(d)  although  simulators  are  enjoying  current  popu- 
larity, due  to  their  flexibility  they  are  expensive  both  in  time  and  cost. 
(22,  6) 

(e)  There  are  also  several  other  drawbacks.  Simula- 
tors .nave  sr.own  mixed  results  for  multiprocessor  and  telecommunication  sys 
terns.  Further,  the  availability  of  language  and  documentation  may  be  dif- 
ficult since  tr.e  packages  do  contain  rroprietary  material.  The  simulator 
costs  which  include  training  to  use  tr.e  language  or  simulator  and  manpower 
and  computer  time  tc  suild  ar.d  run  tr.e  models  may  prove  too  great  to  war- 
rant simulation.  Finally,  tr.e  experience  of  several  users  has  indicated 
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that  bic.uluti.on  cannot  be  used  to  maice  accurate  comparisons.  Allowances 
should  be  made  for  error  bounds  of  perhaps  thirty  percent.  (26,  2CC) 
since  simulators  are  predictive  in  nature,  their  output  must  be  validated 
and  verified,  depending  on  the  complexity  of  the  simulator,  this  process 
may  prove  to  be  overwhelming. 
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* . 7 selection 

As  soor.  as  tne  proposals  have  been  evaluated  and  tne  bencnmsrks  have 
been  completed,  tne  user  must  make  a selection  of  tnat  computer  system 
wnicn  best  meets  r.is  needs  and  yet  is  cost  effective.  Seveial  methods  of 
system  selection  have  been  proposed. 

*.7.1  Tne  iieirhts  and  Scores  Approach 

In  this  approach  the  cna racteristics  cf  the  proposed  systems 
are  divided  into  major  classes  sucn  as  equipment  characteristics,  program- 
mi  ng,  soft  were  support,  pricing,  etc.  Subclasses  are  then  established  with 
in  each  class.  This  division  continues  until  tne  detail  is  as  fine  as  de- 
sired. The  result  is  a tree  structure  which  completely  describes  the  de- 
sired system.  This  general  system  is  tnen  weighted  according  to  the  impor- 
tance tne  user  places  on  each  node  or  subnode  in  the  tree.  For  example,  th 
user  may  decide  tnat  to  meet  nis  needs  he  should  represent  tne  central  pro- 
cessing unit  as  being  twice  as  important  as  memory.  He  may  further  rank 
memory  as  being  as  important  as  the  l/C  interfaces.  He  may  then,  as  an 
example,  weigh  tnose  cr.a  ra  cteri  sti  cs  as  2,  1,  and  1,  respectively.  (26, 
21C)  Since  the  aforementioned  items  are  part  of  the  hardware,  a weirht  is 
assigned  to  tnat  category  in  relation  to  software,  expansion  potential,  etc 
The  scoring  uspect  cf  this  method  is  achieved  by  evaluating  er=  ch  vender's 
proposal  in  the  above  manner.  An  analyst  adds  up  all  the  scores  for  the 
various  brancr.es  and  nodes.  Tnis  process  continues  until  tne  entire  sys- 
tem has  been  evaluated  and  a single  score  for  eacr.  system  has  been  deter- 
mined. The  hignest  score  is  ostensibly  cuosen  tne  winner. 

Altnougn  tnis  approacn  is  complete  and  effective  several  defi- 
ciencies are  . ui te  evident. 
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a.  Ine  method  cf  assigning  weights  to  tne  various  components 
is  subjective  and  depends  on  tne  individual's  understanding-  of  the  system, 
its  requirements,  the  needed  growth  potential,  plus  a host  of  other  vari- 
ables . 

b.  The  entire  evaluation  scheme  is  threatened  if  a particular 
piece  of  equipment  is  not  bid.  This  problem  arises  since  each  node  (twig) 
is  dependent  on  tne  previous  node. 

c.  Anotner  major  deficiency  of  this  method  is  its  inability  to 
satisfactorily  handle,  incorporate,  ar.d  evaluate  the  cost  cf  the  system. 

*.7«<2  Ccst-Value  Techniques 

aince  cost  is  among  tne  major  factors  to  be  considered  in  com- 
puter systems  acquisition,  a "cost-value11  technique  has  been  developed  by 
Dr.  mdward  C.  Jcslin.  This  metnod,  described  in  detail  below,  is  currently 
being  used  by  both  tne  Isavy  ar.d  Air  Force. 

A’hat  is  the  cost-value  technique?  essentially,  it  is  a meth- 
odology wn.icn  recognizes  tne  necessity  cf  evaluating  the  CSR  features  cf 
the  system  and  their  cost  as  offered  by  competing  vendors.  Those  features 
which  have  been  prescribed  as  mandatory  are  not  evaluated;  rather,  they  are 
validated.  The  payments  associated  with  mandatory  requirements  are  rela- 
tively easy  to  identify,  since  tne  vendor  itemizes  them  for  tne  user. 

Since  all  vendors  must  comply  with  the  mandatory  conditions  stated  as  Ven- 
dor Requirements,  wnicn  by  definition  can  only  be  satisfied  by  the  vendor, 
one  must  then  accept  tne  char~es  wnicn  tne  vendors  attacn  to  fulfilling 
tnose  requirements . a vendor's  ebility  to  meet  the  mandatory  requirements 
snould  not  require  evaluation,  since  tne  vender  should  not  nave  submitted  a 
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bid  unless  he  was  able  to  meet  the  requirements.  however,  the  decree  to 
wnich  a system  or  vendor  exceeds  the  minimum  requirements  is  evaluated,  if 
this  excess  is  listed  as  desirable.  (15?  571)  The  cost-value  technique 
offers  two  di sti ngui shine  features}  (1)  it  enables  the  user  to  evaluate  any 
extra  features  and  determine  whether  these  features  are  important  in  them- 
selves or  if  they  are  merely  incidental  to  the  proposed  system;  (2)  it 
enables  tne  user  to  assign  a dollar  value  to  the  OSR . Although  the  dollar 
value  assigned  to  a feature  may  still  be  an  arbitrary  cnoice  made  by  the 
user,  it  offers  a basis  for  comparison  which  can  be  changed  independent  of 
all  other  individually  assigned  values. 

Tne  first  questions  to  be  asiced  are:  ...net  is  and  what  is  not 

considered  to  be  an  CiR  feature?  hhat  cost-value  can  be  assigned  to  it? 

To  avoid  any  bias,  or  appearance  of  bias,  on  tne  part  of  the  evaluation, 
this  study  must  be  initiated  before  tne  proposals  are  received.  It  thus 
becomes  necessary  to  deal  with  hypothetical  or  realistically  anticipated 
extras.  (15,  571)  Ih  order  to  evaluate  the  expected  "extras,"  categories 
such  as  cost,  equipment  characteristics,  expansion  potential,  and  vendor's 
support  of  the  system  are  established.  Should  the  vendor  include  an  item 
wnich  does  not  fall  into  any  one  of  tne  categories  which  the  user  has  estab- 
iis.ned,  a cost-value  template  can  be  establisned  at  that  point  if  the  new 
item  i 3 considered  to  be  important  enough. 

Some  general  co_ments  are  provided  on  the  four  categories  listed 

a bove : 
t 

a.  Cost:  The  cost  must  be  spread  proportionately  ever  the 

expected  life  of  the  system,  end  tne  system  costs  must  change  to  reflect  the 
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costs  of  any  planned  system  expansion  ....  No  cost  item  should  be 
duplicative.  (15,  572) 

b.  Equipment  Cnfl ra cteristi cs : The  cost-value  technique 

does  not  consider  any  equipment  characteristics,  in  tnsmselves,  to  be  im- 
portant extras.  Instead,  their  significance  is  measured  in  teres  of  the 
running  time  of  tne  system  Which  in  turn  deter:  in.es  tne  system's  cost  and 
expansion  potential.  (15,  272)  Cnaracteristics  wnicr.  fall  into  this  group- 
ing may  be  Hardware  compatibility,  reliability,  capacity,  speed,  etc. 

c.  expansion  Potential:  In  order  to  evaluate  tne  expan- 

sion potential  of  a system,  it  is  necessary  to  calculate  tne  running  time 
required  by  tne  system  to  complete  all  of  the  required  applications.  Cr.e 
must  also  knew  the  capacity  of  the  central  processor  and,  finally,  be  able 
to  evaluate  the  special  features  suers  as  buffering  and  parallel  processing-. 

(15,  275) 

d.  System  Support:  The  cost-value  technique  considers  the 

value  of  the  extras  offered  by  each  vendor.  Joslin  sugrests  tnat  tne  sim- 
plest end  perhaps  the  best  method  of  cost-value  assignment  is  to  simply  re- 
quest tne  otner  vendors  to  quote  tne  costs  associated  with  supplying  a ser- 
vice. Thus,  if  one  vendor  offers  2A  hours  on-site  maintenance,  ar.d  other 
vendors  do  not,  it  night  prove  meaningful  to  ask  the  other  vendors  what  the 
extra  charge  would  be.  (15,  272) 

e.  Ctaer  oxtras:  This  category  is  designed  to  deal  with 

those  items  wnicn  do  not  fit  into  any  of  tne  otner  four  previously  mentioned 
groupB . 

After  grouping  all  of  tne  expected  extras  or  Cop's,  value 
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statements  must  then  be  prepared  which  lists 

a.  Tne  C5R  feature--a  detailed  statement  of  the  item  tc  be 

considered . 

b.  Rationale — a detailed  statement  which  snows  tne  metr.ods 
used  to  arrive  at  the  value  assigned  to  tne  feature.  The  value  assigned 
should  be  considered  from  four  viewpoints: 

(1)  The  cost  to  tne  organization  in  teres  of  manpower, 
equipment,  etc.,  of  having  to  do  without  this  Cun. 

(2)  The  cost  of  obtaining  this  OCR  by  in-house  pro- 
gramming, cost  of  extra  equipment,  etc. 

(5)  Tr.e  cost  of  purc.nasinr  this  CSR  from  ^-meone 
other  tnen  the  vendor. 

c.  Tne  templet — a statement  which  shows  hew  the  value  will 
be  awarded  to  varying  amounts  of  the  item  made  available  by  the  vendor. 

d.  Cost  Assessment--this  is  a statement  added  after  tne 
evaluation  which  describes  how  a cost  was  assessed  to  the  various  bidding 
vendors  for  the  Other  Systems  Requirement(s) . 

Value  statements  are  cf  tnree  basic  types  and  are  assessed 

accordingly. 

(1)  Statements  of  those  iters  having  a lo~icsl  maxi- 
mum value  (or  assessable  cost).  Items  of  this  nature  are  the  Ctr.er  Systems 
Requirements  wi.icn  are  available  from  an  independent  source,  i.e.,  soft- 
ware packages,  support,  etc.  If  a vendor  were  to  cnarre  an  amount  in  ex- 
cess of  tne  stited  value,  the  item  would  be  purcr.ased  instead  from  tr.e 
alternate  source  Tat  least  fer  evaluation  purposes).  Tr.erefore,  the  high- 
est assessable  cost  for  those  requi re  rents  is  tne  cost  eta  ted  in  the 


templets  for  these  items  must  identify  the  cost  by  describing  them  as  the 
maximum  cost. 

(2)  statements  of  other  items  having  an  approximate 
maximum  assessable  cost.  Items  of  tnis  nature  ure  Otr.er  Systems  Require- 
ments which  are  largely  made  up  of  fixed  or  predetermine ble  costs,  with 
some  cost  elements  which  are  dependent  upon  the  systems  bid.  An  example 
of  this  type  might  be  the  estimated  developmental  cost  for  special  software, 
where,  if  the  user  were  to  do  it,  it  would  involve  manpower  (predetermir.able 
cost)  and  equipment  (dependent  cost).  Items  of  this  type  are  identified  by 
stating  tne  approximate  cost  (value)  whicn  could  be  assessed  for  failure  to 
bid  tna t item. 

(J)  statements  of  a third  type  involve  an  item  which 
owes  its  value  strictly  to  cost  avoidance  issues.  Items  of  this  type  are: 
location  of  training  or  test  facilities,  space  requirements,  etc.  Approx- 
imate values  are  given  for  tnese  items  also,  however,  the  cost  assessment 
becomes  whatever  t.oey  truly  cost. 

An  example  taken  from  Mr.  Jcsli.n's  book,  Analysis.  Design, 
and  Selection  of  Computer  bystems  illustrates  tne  above  points.  (I3f,  2C) 

(1)  Desirable  Features:  Accountinr  Report  - Opera- 

ting system  capabi.ity  for  maintaining  records  of  time  charges  with  respect 
to  users  and  job  identi  fi  ca  tier,  codes. 

(2)  Rationale:  In-house  development  of  an  accounting 

report  program  would  require  an  estimated  one  man-year  of  systems  software 
analyst  effort  (3ZC,CCC);  one  mar.-year  of  programmer  effort  (52C,CCC),  and 
36,CCC  in  computer  time.  Thus,  the  total  costs  are  34c,CCC.  The  develop- 
ment time  would  range  from  twelve  to  ei-r.teen  montns.  Development  of  tne 


program  using  contractual  programming  support  would  cost  an  estimated 
^f'fCCC  and  consume  twelve  months  time.  Therefore,  a value  of  -ihc,CCC 
will  be  used. 

(3)  Templet:  « maximum  value  of  S^3,CCC  could  be 

assigned  for  the  desired  package.  Value  will  be  assigned,  however,  in 
conjunction  with  a qualitative  evaluation  of  t.oe  fulfillment  of  the  require- 
ments by  each  vendor's  package. 

In  order  to  properly  use  tne  Cost  Value  Technique  for  eval- 
uation, it  is  necessary  to  cring  together  all  of  the  mandatory  requirements 
payments  and  Ctner  systems  Requirements  payments  over  tne  entire  system 
life.  In  order  to  do  this,  four  additional  points  must  be  considered: 
system  life,  present  value,  residual  value,  and  tne  various  procurement 
methods  available,  lacn  of  these  items  were  discussed  previously. 

3*7*3  Contract  Award 

Proposals  meeting  tne  mandatory  requirements  and  complying  with 
the  provisions  of  the  contract  will  be  evaluated  and  award  made  to  that  re- 
sponsible offeror  whose  proposal  is  determined  to  be  tne  lowest  overall 
cost  to  the  Government,  price,  and  other  factors  considered  for  tr.e  sy;tem(s) 
life.  Cost  to  the  Government  includes  t..e  offeror's  prices  (equipment, 
software,  and  support)  over  the  systems  life,  assessments  for  desirable 
feutures  not  satisfactorily  proposed  and  any  predetermined  in-house  expenses 
for  AuPm  installation  and  operation.  Residual  value  will  be  evaluated  for 
any  system(s)  wnere  ownership  resides  with  t.ue  Government. 


ChAPTDA  IV 


phsuftmsu  it X snl.* 


:i::; 


There  are  no  simple,  precise,  quantitative  treasures  by  which  one  ma- 
chine can  be  said  to  be  better  tna r.  anotr.er.  .->1  thougn  there  is  still  a 
large  amount  of  subjectivity  involved,  real  progress  has  been  made  in  many 
facets  of  tne  selection  procoas.  fast  experience  has  shown  that  wall 
charts  and  grapns  were  inadequate.  Progress  has  been  trade  in  assisting 
tne  systems  analyst  ay  various  automated  teenniques.  Similar  pro-ress  can 
be  seen  ir.  m.cuine  evaluation  teenniques.  Instruction  cycles  ar.d  add-times 
ware  improved  upon  by  the  kernel  method,  uitr.  tne  advent  of  the  multi - 
programming/mul tiprccessing  macnine,  tne  evaluation  process  was  modified 
and  cnanged  to  include  live  and  artificial  benenms rks , simulation,  ar.d 
syntiietic  programs. 

I he  future  seems  to  held  some  promising  results  in  store  for  tne 
evaluator.  Task  Group  1J,  Federal  Information  Processing  standards,  has 
released  some  guidelines  cn  standard  benchmarks.  A new  system  known  as 
Information  oystem  Design  and  Optimization  by-stem  (iSDCd)  is  being  devel- 
oped . (6,  lr2)  This  effort  is  striving  to  analyze  the  system  as  a whole — 
tne  organization  as  a single  unit  rat.-.er  t/.an  Just  t.ie  various  parts. 

As  computer  applications  are  being  integrated,  techniques  for  escr.  cf 
the  rnases  of  system  development  are  oeine  integrated,  n natural  extension 
of  computerized  prooiem  statements  is  translation  of  tnose  statements  into 
programming  language  statements.  Tr.e  i«DC«  project  is  designed  to  produce 
such  a by. tern.  ..nile  completion  of  tne  IbDCo  project  is  some  time  away,  a 
sufficient  numser  of  moduies  have  been  desi gr.ed  and  tested  to  prove  tne 
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validity  of  the  appro.-* oh. 

IoDOj  consists  of  four  mcd-iles.  The  Lata  hecr  r~  nizer  accepts:  (1' 

specifications  for  tne  desired  stcr-ge  structures  from  the  physical  systems 
design  process;  (2)  defir.iti  r.  ::  ; ; t-.  as  summarized  by  tr.e  Problem  State- 
ment Analyzer;  (5;  t:.e  specifications  of  the  hardware  to  be  used;  and  (^) 
tne  data  as  currently  exists  and  i.ts  storage  structure.  It  then  stores  the 
data  on  the  selected  devices  in  the  form  specified.  The  third  module,  the 
Code  Generator,  accepts  specifications  from  the  physical  design  process  and 
organizes  the  problem  statements  into  prcrrams  recognizing  the  data  inter- 
face as  specified  by  tne  Lata  heorga n.izer . The  code  prcdaced  may  be  eitr.er 
mac.uine  code,  statements  in  a higher-level  language  (e.g.,  Cobol),  or  para- 
meters to  a software  package.  These  two  modules  perform,  automatically, 
the  functioning  of  programming  ur.d  file  construction.  Tne  final  module  of 
the  IwLC-  system  is  the  _y_ terns  Lirectrr.  It  accepts  tne  cede  generated, 
tne  timing  specifications  c s determined  bp-  the  physical  design  alrorithm, 
and  the  specifications  from  tne  Lata  Reorge  • izer,  ar.d  produces  tne  turret 
information  system.  This  system  is  now  ready  to  accept  inputs  frem  the 
environment  and  produce  tne  necessary  outputs  according  to  the  requirements 
expressed  in  tu«  problem  statement. 

t-'ji ’or  projects  such  as  t.nese  will  allow  simpler  and  clearer  analyses. 
Ihese  methods,  in  turn,  will  allow  clearer  evaluation  techniques  to  be 
developed  . 
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CONCLUSIONS  COUNTS 

Tne  basic  research  question  t.nat  must  be  answered  is  es  fellows: 

..mat  methods  are  available  to  a manager  in  t;.e  evaluation  and  selection  of 
a computer  system  or  parts  thereof? 

It  appears  tnat  the  major  emphasis  which  a manager  must  place  in  the 
overall  process  is  on  a modification  in  tne  initial  analysis  time.  In 
order  to  lessen  tne  subjectivism,  wnic.n  is  already  a part  of  any  evaluation, 
and  to  lessen  tne  criticism  of  "representativeness,*  considerably  more  time 
and  effort  must  be  expended  at  tne  "front  end,"  of  any  systems  development, 
mmphasis  must  be  placed  on  data  gathering  and  analysis  of  tne  present  system 
or  tne  proposed  system.  Tecnniques  nave  been  snd  are  being  developed  which 
will  aid  tne  analyst  in  his  work.  ISDCS  was  already  mentioned  as  tne  cur- 
rent effort  in  systems  design.  Hany  of  the  commercially  available  simula- 
tors must  be  used  in  order  to  evaluate  a users  present  system,  as  well  as 
the  vendor’s  proposed  systems.  Conceptual  systems  do  not  lend  themselves 
to  the  present  conventional  benenmarks.  Simulation  and  synthetic  programs, 
wnich  are  representative  of  tne  expected  process,  are  two  alternatives 
tnat  are  suggested  for  conceptual  systems.  Tnis  is  recommended  in  spite 
of  the  questionable  accuracy  of  simulation  and  t:.e  current  nonportability 
of  many  synthetic  programs.  One  other  alternative  must  be  mentioned, 
iencnm&rks  snould  be  run  on  systems  wnicn  are  similar  to  the  ones  wnich  are 
expected  to  be  bid.  Altnougr.  tnis  is  not  sc  satisfactory  us  an  actual  run 
on  the  bidders  equipment,  a rougn  estimate  can  be  obtained  from  tale 
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experience.  Finally,  tne  system  will  perform  as  expected  if  time  and  care 
are  taken  in  tna  analysis  efforts  as  well  as  a careful  usage  of  the  various 
evaluation  methods. 

Tnrougnout  this  rs.ort  I have  attempted  to  maintain  a general  approach 
to  tne  problem.  Computers,  however,  are  used  in  almost  3ll  programs  yet 
they  continue  to  receive  inadequate  attention,  aitness  the  tragic  failure 
of  the  U.  S.  Air  Force  logistics  computer  system  which  was  scrapped  after 
9 years  of  effort  and  i25C  million.  Its  failure  was  attributed  to  many 
causes,  some  of  which  was  the  aspect  of  systems  specifications,  testing, 
and  computer  acquisition.  Any  program  manager  (?K;  or  similsr  individual 
who  is  charged  with  development  of  a system,  be  it  a weapons  system  or  a 
computer  system,  must  utilize  many  of  tne  areas  contained  in  this  paper. 

In  essence,  a variety  of  subjects  have  been  presented  from  wnicn  a diligent 
Pn  car.  pick  and  cnoose  as  tne  situation  and  applicable  procedures  allow. 
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